[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:36 dist-test-slave-djjw krb5kdc[4088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:36 dist-test-slave-djjw krb5kdc[4088](info): set up 2 sockets
Sep 28 10:57:36 dist-test-slave-djjw krb5kdc[4088](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 10:57:37.147758 4081 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 10:57:37.457757 4081 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.310s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 10:57:37.688690 4081 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.231s user 0.003s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 10:57:37.909332 4081 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.220s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/test-user.keytab.
W20260928 10:57:38.693605 4081 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.784s user 0.004s sys 0.000s
Sep 28 10:57:38 dist-test-slave-djjw krb5kdc[4088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
W20260928 10:57:39.923131 4081 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.252.126: real 1.187s user 0.000s sys 0.008s
I20260928 10:57:39.924736 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:34511
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:40.255328 4101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:40.255635 4101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:40.255764 4101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:40.263115 4101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:40.263235 4101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:40.263288 4101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:40.263366 4101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:40.263425 4101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:40.273445 4101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:34511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:40.274819 4101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:40.276198 4101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:40.288339 4109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:40.288494 4107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:40.288339 4106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:40.288734 4101 server_base.cc:1034] running on GCE node
I20260928 10:57:40.289871 4101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:40.291514 4101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:40.292725 4101 hybrid_clock.cc:648] HybridClock initialized: now 1790593060292673 us; error 46 us; skew 500 ppm
I20260928 10:57:40.293090 4101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:40 dist-test-slave-djjw krb5kdc[4088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:40.296222 4101 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:57:40.297972 4101 webserver.cc:460] Webserver started at http://127.3.252.126:34881/ using document root <none> and password file <none>
I20260928 10:57:40.298714 4101 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:40.298826 4101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:40.299153 4101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:40.302361 4101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "bf040c6df32a4195837c15c9d385dda3"
format_stamp: "Formatted at 2026-09-28 10:57:40 on dist-test-slave-djjw"
server_key: "702a7516e4c3aa40431747666415cf50"
server_key_iv: "9a01a677b33c88c51d908aebafc3950e"
server_key_version: "encryptionkey@0"
I20260928 10:57:40.303162 4101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "bf040c6df32a4195837c15c9d385dda3"
format_stamp: "Formatted at 2026-09-28 10:57:40 on dist-test-slave-djjw"
server_key: "702a7516e4c3aa40431747666415cf50"
server_key_iv: "9a01a677b33c88c51d908aebafc3950e"
server_key_version: "encryptionkey@0"
I20260928 10:57:40.309152 4101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 10:57:40.312923 4116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:40.314563 4101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:57:40.315342 4101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "bf040c6df32a4195837c15c9d385dda3"
format_stamp: "Formatted at 2026-09-28 10:57:40 on dist-test-slave-djjw"
server_key: "702a7516e4c3aa40431747666415cf50"
server_key_iv: "9a01a677b33c88c51d908aebafc3950e"
server_key_version: "encryptionkey@0"
I20260928 10:57:40.315552 4101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:40.331699 4101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:40.336375 4101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:40.336632 4101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:40.354554 4101 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:34511
I20260928 10:57:40.354568 4167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:34511 every 8 connection(s)
I20260928 10:57:40.355983 4101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:40.359870 4168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:40.360653 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4101
I20260928 10:57:40.360929 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:57:40.362880 4081 external_mini_cluster.cc:1428] Setting key 5a005f3ccee9806a693d6d4c4e3fe57a
I20260928 10:57:40.371377 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3: Bootstrap starting.
I20260928 10:57:40.374419 4168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:40.375463 4168 log.cc:824] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:40.378373 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3: No bootstrap required, opened a new log
I20260928 10:57:40.385690 4168 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf040c6df32a4195837c15c9d385dda3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34511 } }
I20260928 10:57:40.386138 4168 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:40.386246 4168 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf040c6df32a4195837c15c9d385dda3, State: Initialized, Role: FOLLOWER
I20260928 10:57:40.386909 4168 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf040c6df32a4195837c15c9d385dda3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34511 } }
I20260928 10:57:40.387138 4168 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:40.387269 4168 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:40.387437 4168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 10:57:40 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:40.400869 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:40.367052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.3.252.126:34511)
0928 10:57:40.367884 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:40.367902 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:40.369111 (+ 1209us) server_negotiation.cc:407] Connection header received
0928 10:57:40.378002 (+ 8891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:40.378026 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:40.378503 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:40.378895 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:40.382630 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:40.384077 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:40.385815 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:40.386693 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:40.386880 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:40.392516 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:40.392526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:40.392543 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:40.392601 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:40.395420 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:40.396265 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:40.396272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:40.396279 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:40.396458 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:40.397273 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:40.397277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:40.397279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:40.397885 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:40.398164 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:40.399151 (+ 987us) server_negotiation.cc:299] Negotiation successful
0928 10:57:40.399583 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":383,"threads_started":1}
I20260928 10:57:40.410722 4168 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf040c6df32a4195837c15c9d385dda3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34511 } }
I20260928 10:57:40.411339 4168 leader_election.cc:302] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf040c6df32a4195837c15c9d385dda3; no voters:
I20260928 10:57:40.412431 4168 leader_election.cc:288] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:40.412550 4173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:40.413884 4173 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [term 1 LEADER]: Becoming Leader. State: Replica: bf040c6df32a4195837c15c9d385dda3, State: Running, Role: LEADER
I20260928 10:57:40.414726 4173 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf040c6df32a4195837c15c9d385dda3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34511 } }
I20260928 10:57:40.414912 4168 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:40.418561 4174 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf040c6df32a4195837c15c9d385dda3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf040c6df32a4195837c15c9d385dda3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34511 } } }
I20260928 10:57:40.418533 4175 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf040c6df32a4195837c15c9d385dda3. Latest consensus state: current_term: 1 leader_uuid: "bf040c6df32a4195837c15c9d385dda3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf040c6df32a4195837c15c9d385dda3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34511 } } }
I20260928 10:57:40.419008 4174 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:40.419009 4175 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:40.420912 4182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:40.425633 4182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:40.435909 4182 catalog_manager.cc:1347] Generated new cluster ID: 416554daf490454b8a11c59d17a0276e
I20260928 10:57:40.436120 4182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:40.445900 4182 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:40.447377 4182 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:40.454159 4182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf040c6df32a4195837c15c9d385dda3: Generated new TSK 0
I20260928 10:57:40.455307 4182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
W20260928 10:57:41.248895 4081 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.252.65: real 0.780s user 0.001s sys 0.009s
I20260928 10:57:41.249887 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:34511
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:41.573587 4194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:41.573894 4194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:41.574029 4194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:41.581100 4194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:41.581204 4194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:41.581360 4194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:57:41.590935 4194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:34511
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:41.592186 4194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:41.593461 4194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:41.605988 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:41.606019 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:41.606467 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:41.606673 4194 server_base.cc:1034] running on GCE node
I20260928 10:57:41.607195 4194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:41.608047 4194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:41.609244 4194 hybrid_clock.cc:648] HybridClock initialized: now 1790593061609191 us; error 42 us; skew 500 ppm
I20260928 10:57:41.609545 4194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:41 dist-test-slave-djjw krb5kdc[4088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:41.616501 4194 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:57:41.618103 4194 webserver.cc:460] Webserver started at http://127.3.252.65:33279/ using document root <none> and password file <none>
I20260928 10:57:41.618842 4194 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:41.618960 4194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:41.619259 4194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:41.622396 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "084ed62d24a34d0ebf47f96f5398edc3"
format_stamp: "Formatted at 2026-09-28 10:57:41 on dist-test-slave-djjw"
server_key: "d0b0879d167f329ba02f18ee00c1bc82"
server_key_iv: "426096e0083e50c0b9d43a9de6a0f02f"
server_key_version: "encryptionkey@0"
I20260928 10:57:41.623159 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "084ed62d24a34d0ebf47f96f5398edc3"
format_stamp: "Formatted at 2026-09-28 10:57:41 on dist-test-slave-djjw"
server_key: "d0b0879d167f329ba02f18ee00c1bc82"
server_key_iv: "426096e0083e50c0b9d43a9de6a0f02f"
server_key_version: "encryptionkey@0"
I20260928 10:57:41.628407 4194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:57:41.631803 4209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:41.633256 4194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:57:41.633417 4194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "084ed62d24a34d0ebf47f96f5398edc3"
format_stamp: "Formatted at 2026-09-28 10:57:41 on dist-test-slave-djjw"
server_key: "d0b0879d167f329ba02f18ee00c1bc82"
server_key_iv: "426096e0083e50c0b9d43a9de6a0f02f"
server_key_version: "encryptionkey@0"
I20260928 10:57:41.633606 4194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:41.664777 4194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:41.668896 4194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:41.669137 4194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:41.670046 4194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:57:41.671274 4194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:41.671351 4194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:41.671433 4194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:41.671482 4194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:41.702687 4194 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:33159
I20260928 10:57:41.702741 4321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:33159 every 8 connection(s)
I20260928 10:57:41.703895 4194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:57:41.705591 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4194
I20260928 10:57:41.705864 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:57:41.706485 4081 external_mini_cluster.cc:1428] Setting key fa9aadb73c5518b18a0532c42aeb96a8
Sep 28 10:57:41 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
I20260928 10:57:41.725984 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:41.706682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:40529 (local address 127.3.252.126:34511)
0928 10:57:41.707197 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:41.707201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:41.709692 (+ 2491us) server_negotiation.cc:407] Connection header received
0928 10:57:41.710446 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:41.710452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:41.710531 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:41.710622 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:41.712248 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:41.712971 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:41.715256 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:41.715645 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:41.715780 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:41.719659 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:41.719665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:41.719668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:41.719706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:41.722803 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:41.723620 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:41.723625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:41.723627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:41.723787 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:41.724257 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:41.724260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:41.724262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:41.724524 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:41.724720 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:41.725367 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 10:57:41.725584 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":250,"threads_started":1}
I20260928 10:57:41.726955 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:41.707662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:34511 (local address 127.3.252.65:40529)
0928 10:57:41.708420 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:41.708470 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:41.710190 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:41.710881 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:41.710890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:41.711403 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:41.712058 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:41.712083 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:41.713113 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:41.713118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:41.715115 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:41.715140 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:41.715874 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:41.715882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:41.716121 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:41.716752 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:41.716772 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:41.719409 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:41.722967 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:41.722974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:41.722989 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:41.723491 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:41.723896 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:41.723899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:41.723902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:41.724145 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:41.724648 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:41.724657 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:41.725019 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 10:57:41.725265 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 10:57:41.725786 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":287,"threads_started":1}
I20260928 10:57:41.728695 4322 heartbeater.cc:347] Connected to a master server at 127.3.252.126:34511
I20260928 10:57:41.728981 4322 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:41.729691 4322 heartbeater.cc:511] Master 127.3.252.126:34511 requested a full tablet report, sending...
I20260928 10:57:41.731809 4133 ts_manager.cc:194] Registered new tserver with Master: 084ed62d24a34d0ebf47f96f5398edc3 (127.3.252.65:33159)
I20260928 10:57:41.733314 4133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:40529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:57:41.743038 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:34511
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:42.067039 4328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:42.067332 4328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:42.067451 4328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:42.074724 4328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:42.074857 4328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:42.075008 4328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:57:42.085543 4328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:34511
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:42.086983 4328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:42.088265 4328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:42.101639 4333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:42.101603 4336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:42.101568 4334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:42.102599 4328 server_base.cc:1034] running on GCE node
I20260928 10:57:42.103223 4328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:42.104136 4328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:42.105335 4328 hybrid_clock.cc:648] HybridClock initialized: now 1790593062105285 us; error 43 us; skew 500 ppm
I20260928 10:57:42.105646 4328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:42.108772 4328 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:57:42.110402 4328 webserver.cc:460] Webserver started at http://127.3.252.66:42015/ using document root <none> and password file <none>
I20260928 10:57:42.111078 4328 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:42.111177 4328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:42.111433 4328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:42.114373 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "04601522d25c4e7bbb7f1703513f9ef5"
format_stamp: "Formatted at 2026-09-28 10:57:42 on dist-test-slave-djjw"
server_key: "4084fc605c2fdb2be98654d5cf25a195"
server_key_iv: "3282faaa1a32c5b65c9a6a1249be6ead"
server_key_version: "encryptionkey@0"
I20260928 10:57:42.115128 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "04601522d25c4e7bbb7f1703513f9ef5"
format_stamp: "Formatted at 2026-09-28 10:57:42 on dist-test-slave-djjw"
server_key: "4084fc605c2fdb2be98654d5cf25a195"
server_key_iv: "3282faaa1a32c5b65c9a6a1249be6ead"
server_key_version: "encryptionkey@0"
I20260928 10:57:42.121135 4328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:57:42.126010 4343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:42.127859 4328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:57:42.128036 4328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "04601522d25c4e7bbb7f1703513f9ef5"
format_stamp: "Formatted at 2026-09-28 10:57:42 on dist-test-slave-djjw"
server_key: "4084fc605c2fdb2be98654d5cf25a195"
server_key_iv: "3282faaa1a32c5b65c9a6a1249be6ead"
server_key_version: "encryptionkey@0"
I20260928 10:57:42.128273 4328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:42.151557 4328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:42.156000 4328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:42.156282 4328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:42.157359 4328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:57:42.158778 4328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:42.158852 4328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:42.158969 4328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:42.159020 4328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:42.191519 4328 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:35351
I20260928 10:57:42.191563 4455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:35351 every 8 connection(s)
I20260928 10:57:42.192776 4328 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:57:42.198758 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4328
I20260928 10:57:42.198935 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:57:42.199334 4081 external_mini_cluster.cc:1428] Setting key 6aaed64a7605f101c3ac7effe50f8bbf
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:42.213130 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.195284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:59437 (local address 127.3.252.126:34511)
0928 10:57:42.195472 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:42.195477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:42.196645 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 10:57:42.198504 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:42.198510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:42.198597 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:42.198683 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:42.200420 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.201235 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.202834 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.203252 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.203361 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.206803 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:42.206808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:42.206811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:42.206846 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:42.209706 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.210729 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.210734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.210737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.210895 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.211372 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.211375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.211377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.211660 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:42.211768 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:42.212689 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 10:57:42.212891 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 10:57:42.214562 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.195806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:34511 (local address 127.3.252.66:59437)
0928 10:57:42.196492 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:42.196530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:42.198219 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:42.198966 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:42.198977 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:42.199552 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:42.200247 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:42.200280 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.201392 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.201396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:42.202697 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:42.202709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.203473 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.203480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:42.203691 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.204305 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:42.204328 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:42.206565 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:42.209901 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:42.209910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:42.209929 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:42.210556 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:42.211006 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:42.211011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:42.211015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:42.211273 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:42.211783 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:42.211792 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:42.212287 (+ 495us) client_negotiation.cc:769] Sending connection context
0928 10:57:42.212611 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 10:57:42.213214 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":251,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
I20260928 10:57:42.215996 4456 heartbeater.cc:347] Connected to a master server at 127.3.252.126:34511
I20260928 10:57:42.216347 4456 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:42.217059 4456 heartbeater.cc:511] Master 127.3.252.126:34511 requested a full tablet report, sending...
I20260928 10:57:42.218638 4133 ts_manager.cc:194] Registered new tserver with Master: 04601522d25c4e7bbb7f1703513f9ef5 (127.3.252.66:35351)
I20260928 10:57:42.219558 4133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:59437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:57:42.236769 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:34511
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:42.569376 4461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:42.569675 4461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:42.569768 4461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:42.577031 4461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:42.577155 4461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:42.577272 4461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:57:42.587389 4461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:34511
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:42.588618 4461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:42.589877 4461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:42.603216 4466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:42.603247 4467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:42.603235 4469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:42.603868 4461 server_base.cc:1034] running on GCE node
I20260928 10:57:42.604460 4461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:42.605387 4461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:42.606575 4461 hybrid_clock.cc:648] HybridClock initialized: now 1790593062606523 us; error 42 us; skew 500 ppm
I20260928 10:57:42.606881 4461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:42.610025 4461 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:57:42.611522 4461 webserver.cc:460] Webserver started at http://127.3.252.67:38127/ using document root <none> and password file <none>
I20260928 10:57:42.612221 4461 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:42.612318 4461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:42.612576 4461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:42.615608 4461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "e57cb60ed6c7464ea9910b56f8fa5e90"
format_stamp: "Formatted at 2026-09-28 10:57:42 on dist-test-slave-djjw"
server_key: "4ab6863621a831d9d39415a4ffe6027c"
server_key_iv: "61359ff45c0e9759ae2420eaf2a83f8c"
server_key_version: "encryptionkey@0"
I20260928 10:57:42.616371 4461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "e57cb60ed6c7464ea9910b56f8fa5e90"
format_stamp: "Formatted at 2026-09-28 10:57:42 on dist-test-slave-djjw"
server_key: "4ab6863621a831d9d39415a4ffe6027c"
server_key_iv: "61359ff45c0e9759ae2420eaf2a83f8c"
server_key_version: "encryptionkey@0"
I20260928 10:57:42.621981 4461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:57:42.626168 4476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:42.627717 4461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:57:42.627887 4461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "e57cb60ed6c7464ea9910b56f8fa5e90"
format_stamp: "Formatted at 2026-09-28 10:57:42 on dist-test-slave-djjw"
server_key: "4ab6863621a831d9d39415a4ffe6027c"
server_key_iv: "61359ff45c0e9759ae2420eaf2a83f8c"
server_key_version: "encryptionkey@0"
I20260928 10:57:42.628088 4461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:42.649683 4461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:42.654021 4461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:42.654295 4461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:42.655274 4461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:57:42.656603 4461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:42.656675 4461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:42.656780 4461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:42.656832 4461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:42.689577 4461 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:39863
I20260928 10:57:42.689620 4588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:39863 every 8 connection(s)
I20260928 10:57:42.690853 4461 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:57:42.693394 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4461
I20260928 10:57:42.693588 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:57:42.694054 4081 external_mini_cluster.cc:1428] Setting key 609cac1c0b821bf3f9be3f8ed5cc2856
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:42.711597 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.693444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:44651 (local address 127.3.252.126:34511)
0928 10:57:42.693646 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:42.693652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:42.694989 (+ 1337us) server_negotiation.cc:407] Connection header received
0928 10:57:42.697197 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:42.697203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:42.697285 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:42.697369 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:42.699445 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.700179 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.701794 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.702399 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.702531 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.706046 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:42.706051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:42.706054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:42.706088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:42.708532 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.709347 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.709352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.709355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.709515 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.710027 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.710030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.710033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.710340 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:42.710495 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:42.711189 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 10:57:42.711357 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:57:42.712580 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.693993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:34511 (local address 127.3.252.67:44651)
0928 10:57:42.694796 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:42.694850 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:42.696938 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:42.697616 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:42.697625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:42.698717 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:42.699283 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:42.699300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.700324 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.700329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:42.701656 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:42.701667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.702665 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.702671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:42.702897 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.703524 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:42.703550 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:42.705799 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:42.708680 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:42.708687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:42.708701 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:42.709236 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:42.709627 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:42.709631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:42.709634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:42.709875 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:42.710474 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:42.710480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:42.710850 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 10:57:42.711111 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 10:57:42.711502 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":289,"threads_started":1}
I20260928 10:57:42.713876 4589 heartbeater.cc:347] Connected to a master server at 127.3.252.126:34511
I20260928 10:57:42.714155 4589 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:42.714857 4589 heartbeater.cc:511] Master 127.3.252.126:34511 requested a full tablet report, sending...
I20260928 10:57:42.716262 4133 ts_manager.cc:194] Registered new tserver with Master: e57cb60ed6c7464ea9910b56f8fa5e90 (127.3.252.67:39863)
I20260928 10:57:42.717041 4133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:44651
I20260928 10:57:42.722569 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:57:42.736176 4322 heartbeater.cc:503] Master 127.3.252.126:34511 was elected leader, sending a full tablet report...
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:57:42.763701 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.749636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.3.252.65:33159)
0928 10:57:42.750256 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:42.750263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:42.750291 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:57:42.750438 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:42.750445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:42.750658 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:42.750794 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:42.752094 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.753080 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.754369 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.754926 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.755025 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.758550 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:42.758562 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:42.758577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:42.758614 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:42.761211 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.761906 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.761913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.761915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.762035 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.762351 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.762353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.762355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.762692 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:42.762933 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:42.763166 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 10:57:42.763362 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":336,"threads_started":1}
W20260928 10:57:42.764972 4237 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57564
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:57:42.806908 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.794075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.3.252.65:33159)
0928 10:57:42.794316 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:42.794323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:42.794494 (+ 171us) server_negotiation.cc:407] Connection header received
0928 10:57:42.794670 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:42.794675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:42.794765 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:42.794852 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:42.796144 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.796843 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.798099 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.798492 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.798600 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.802181 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:42.802186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:42.802189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:42.802221 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:42.804586 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.805176 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.805180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.805183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.805293 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.805622 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.805626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.805628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.805896 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:42.806110 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:42.806415 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 10:57:42.806617 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 10:57:42.808282 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4194
I20260928 10:57:42.820318 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.809085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.3.252.65:33159)
0928 10:57:42.809235 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:42.809240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:42.809507 (+ 267us) server_negotiation.cc:407] Connection header received
0928 10:57:42.809639 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:42.809644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:42.809699 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:42.809772 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:42.810855 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.811512 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.812746 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.813573 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.813681 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.815762 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:42.815766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:42.815769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:42.815797 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:42.817971 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.818527 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.818533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.818536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.818657 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.819015 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.819020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.819044 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.819280 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:42.819518 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:42.819781 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 10:57:42.819986 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 10:57:42.820958 4317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:42.901305 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4194
I20260928 10:57:42.914615 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4328
Sep 28 10:57:42 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:57:42.929442 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:42.915472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.3.252.66:35351)
0928 10:57:42.915940 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:42.915947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:42.915971 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:57:42.916035 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:42.916042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:42.916239 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:42.916391 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:42.917531 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.918536 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.919775 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:42.920433 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:42.920538 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:42.924431 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:42.924444 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:42.924459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:42.924494 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:42.927018 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.927666 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.927672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.927675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.927788 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:42.928082 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:42.928085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:42.928087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:42.928431 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:42.928650 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:42.928955 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 10:57:42.929142 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":247,"threads_started":1}
I20260928 10:57:42.930858 4451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:43.012130 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4328
I20260928 10:57:43.030340 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4461
Sep 28 10:57:43 dist-test-slave-djjw krb5kdc[4088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:57:43.046208 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:43.031413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.3.252.67:39863)
0928 10:57:43.032009 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:43.032016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:43.032044 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:57:43.032105 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:43.032111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:43.032314 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:43.032493 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:43.033709 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:43.034798 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:43.036088 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:43.036668 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:43.036780 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:43.040807 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:43.040818 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:43.040835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:43.040872 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:43.043518 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:43.044167 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:43.044174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:43.044177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:43.044299 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:43.044629 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:43.044632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:43.044635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:43.044965 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:43.045253 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:43.045547 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 10:57:43.045798 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":292,"threads_started":1}
I20260928 10:57:43.047449 4584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:43.129976 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4461
I20260928 10:57:43.143393 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4101
I20260928 10:57:43.144492 4163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:43.245805 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4101
2026-09-28T10:57:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (7181 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:43 dist-test-slave-djjw krb5kdc[4610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:43 dist-test-slave-djjw krb5kdc[4610](info): set up 2 sockets
Sep 28 10:57:43 dist-test-slave-djjw krb5kdc[4610](info): commencing operation
krb5kdc: starting...
W20260928 10:57:44.237936 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:57:44 dist-test-slave-djjw krb5kdc[4610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:57:44.409657 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:43035
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:44.725204 4623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:44.725499 4623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:44.725587 4623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:44.732897 4623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:44.733019 4623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:44.733062 4623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:44.733112 4623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:44.733153 4623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:44.743140 4623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:44.744411 4623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:44.745648 4623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:44.757227 4628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:44.757158 4629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:44.757184 4631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:44.757951 4623 server_base.cc:1034] running on GCE node
I20260928 10:57:44.758744 4623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:44.760210 4623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:44.761413 4623 hybrid_clock.cc:648] HybridClock initialized: now 1790593064761356 us; error 53 us; skew 500 ppm
I20260928 10:57:44.761751 4623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:44 dist-test-slave-djjw krb5kdc[4610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593064, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:44.764773 4623 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:57:44.766438 4623 webserver.cc:460] Webserver started at http://127.3.252.126:46645/ using document root <none> and password file <none>
I20260928 10:57:44.767127 4623 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:44.767227 4623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:44.767510 4623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:44.770550 4623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e"
format_stamp: "Formatted at 2026-09-28 10:57:44 on dist-test-slave-djjw"
server_key: "7da650324573d7dc941da0d56cd996a8"
server_key_iv: "932ccf7a94300a5af5989c97d7c033f3"
server_key_version: "encryptionkey@0"
I20260928 10:57:44.771302 4623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e"
format_stamp: "Formatted at 2026-09-28 10:57:44 on dist-test-slave-djjw"
server_key: "7da650324573d7dc941da0d56cd996a8"
server_key_iv: "932ccf7a94300a5af5989c97d7c033f3"
server_key_version: "encryptionkey@0"
I20260928 10:57:44.776600 4623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:57:44.780151 4638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:44.781598 4623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:57:44.781760 4623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e"
format_stamp: "Formatted at 2026-09-28 10:57:44 on dist-test-slave-djjw"
server_key: "7da650324573d7dc941da0d56cd996a8"
server_key_iv: "932ccf7a94300a5af5989c97d7c033f3"
server_key_version: "encryptionkey@0"
I20260928 10:57:44.781993 4623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:44.799064 4623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:44.803213 4623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:44.803479 4623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:44.820622 4623 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:43035
I20260928 10:57:44.820624 4689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:43035 every 8 connection(s)
I20260928 10:57:44.822052 4623 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:44.825901 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4623
I20260928 10:57:44.826020 4690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:44.826104 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:57:44.826733 4081 external_mini_cluster.cc:1428] Setting key 578c7a186f59fdf6be378aff46f3bc82
I20260928 10:57:44.840068 4690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e: Bootstrap starting.
Sep 28 10:57:44 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:44.843719 4690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:44.844805 4690 log.cc:824] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:44.847718 4690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e: No bootstrap required, opened a new log
I20260928 10:57:44.850255 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:44.828699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.3.252.126:43035)
0928 10:57:44.829463 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:44.829475 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:44.829516 (+ 41us) server_negotiation.cc:407] Connection header received
0928 10:57:44.831865 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:44.831902 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:44.832358 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:44.832735 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:44.834625 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:44.836007 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:44.837796 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:44.838554 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:44.838703 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:44.843166 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:44.843174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:44.843192 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:44.843235 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:44.846203 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:44.846889 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:44.846897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:44.846903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:44.847061 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:44.847433 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:44.847436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:44.847438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:44.847973 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:44.848238 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:44.848709 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 10:57:44.849122 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":353,"threads_started":1}
I20260928 10:57:44.855593 4690 raft_consensus.cc:348] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43035 } }
I20260928 10:57:44.856009 4690 raft_consensus.cc:374] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:44.856104 4690 raft_consensus.cc:729] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18f4f7cba1549cbaf59ddfdee3cce1e, State: Initialized, Role: FOLLOWER
I20260928 10:57:44.856681 4690 consensus_queue.cc:260] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43035 } }
I20260928 10:57:44.856905 4690 raft_consensus.cc:388] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:44.857030 4690 raft_consensus.cc:482] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:44.857179 4690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:44.859727 4690 raft_consensus.cc:504] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43035 } }
I20260928 10:57:44.860244 4690 leader_election.cc:302] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a18f4f7cba1549cbaf59ddfdee3cce1e; no voters:
I20260928 10:57:44.860829 4690 leader_election.cc:288] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:44.860937 4695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:44.861536 4695 raft_consensus.cc:686] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [term 1 LEADER]: Becoming Leader. State: Replica: a18f4f7cba1549cbaf59ddfdee3cce1e, State: Running, Role: LEADER
I20260928 10:57:44.862082 4695 consensus_queue.cc:237] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43035 } }
I20260928 10:57:44.862913 4690 sys_catalog.cc:564] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:44.865118 4697 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a18f4f7cba1549cbaf59ddfdee3cce1e. Latest consensus state: current_term: 1 leader_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43035 } } }
I20260928 10:57:44.865160 4696 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18f4f7cba1549cbaf59ddfdee3cce1e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43035 } } }
I20260928 10:57:44.865512 4697 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:44.865510 4696 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:44.867715 4702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:44.872685 4702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:44.882256 4702 catalog_manager.cc:1347] Generated new cluster ID: 902093e95b4146c08466c271769476da
I20260928 10:57:44.882386 4702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:44.899822 4702 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:44.901193 4702 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:44.909245 4702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a18f4f7cba1549cbaf59ddfdee3cce1e: Generated new TSK 0
I20260928 10:57:44.910192 4702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:57:44.960040 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43035
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:45.288906 4716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:45.289204 4716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:45.289330 4716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:45.298035 4716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:45.298187 4716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:45.298331 4716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:57:45.309419 4716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:45.310819 4716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:45.312265 4716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:45.326341 4721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:45.326316 4724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:45.326367 4722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:45.327229 4716 server_base.cc:1034] running on GCE node
I20260928 10:57:45.327893 4716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:45.328927 4716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:45.330138 4716 hybrid_clock.cc:648] HybridClock initialized: now 1790593065330083 us; error 48 us; skew 500 ppm
I20260928 10:57:45.330468 4716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:45 dist-test-slave-djjw krb5kdc[4610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:45.333602 4716 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:57:45.335191 4716 webserver.cc:460] Webserver started at http://127.3.252.65:42625/ using document root <none> and password file <none>
I20260928 10:57:45.335911 4716 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:45.336035 4716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:45.336334 4716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:45.339663 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "19b6c3a6dd2e409abac9c5bf6175d62b"
format_stamp: "Formatted at 2026-09-28 10:57:45 on dist-test-slave-djjw"
server_key: "4570baa7567ecfc72eb7d38bfb89bb69"
server_key_iv: "cd41d65b5bdb129e7eae62393c5fb81c"
server_key_version: "encryptionkey@0"
I20260928 10:57:45.340538 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "19b6c3a6dd2e409abac9c5bf6175d62b"
format_stamp: "Formatted at 2026-09-28 10:57:45 on dist-test-slave-djjw"
server_key: "4570baa7567ecfc72eb7d38bfb89bb69"
server_key_iv: "cd41d65b5bdb129e7eae62393c5fb81c"
server_key_version: "encryptionkey@0"
I20260928 10:57:45.346675 4716 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260928 10:57:45.350626 4731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:45.352149 4716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:57:45.352331 4716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "19b6c3a6dd2e409abac9c5bf6175d62b"
format_stamp: "Formatted at 2026-09-28 10:57:45 on dist-test-slave-djjw"
server_key: "4570baa7567ecfc72eb7d38bfb89bb69"
server_key_iv: "cd41d65b5bdb129e7eae62393c5fb81c"
server_key_version: "encryptionkey@0"
I20260928 10:57:45.352551 4716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:45.374953 4716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:45.379468 4716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:45.379770 4716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:45.380755 4716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:57:45.382215 4716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:45.382292 4716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:45.382424 4716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:45.382478 4716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:45.413286 4716 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:38193
I20260928 10:57:45.413352 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:38193 every 8 connection(s)
I20260928 10:57:45.414578 4716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:57:45.418982 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4716
I20260928 10:57:45.419175 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:57:45.419746 4081 external_mini_cluster.cc:1428] Setting key 6f5a908d7c54e5ed049df9a1d1a39143
Sep 28 10:57:45 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 10:57:45.434937 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:45.417103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:37881 (local address 127.3.252.126:43035)
0928 10:57:45.417528 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:45.417533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:45.418632 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 10:57:45.420550 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:45.420556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:45.420639 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:45.420731 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:45.422293 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.423020 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.424552 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.425001 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.425130 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:45.428685 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:45.428692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:45.428696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:45.428737 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:45.431847 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:45.432579 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:45.432585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:45.432589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:45.432728 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:45.433140 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:45.433145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:45.433149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:45.433501 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:45.433649 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:45.434418 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 10:57:45.434650 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":224,"threads_started":1}
I20260928 10:57:45.435868 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:45.417678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:43035 (local address 127.3.252.65:37881)
0928 10:57:45.418442 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:45.418484 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:45.420272 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:45.420990 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:45.420999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:45.421507 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:45.422136 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:45.422153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.423172 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.423175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:45.424418 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:45.424430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.425217 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.425223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:45.425439 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:45.426140 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:45.426166 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:45.428443 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:45.431999 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:45.432006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:45.432021 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:45.432448 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:45.432820 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:45.432823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:45.432825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:45.433026 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:45.433606 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:45.433612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:45.434030 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 10:57:45.434258 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 10:57:45.434694 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":310,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
I20260928 10:57:45.437439 4844 heartbeater.cc:347] Connected to a master server at 127.3.252.126:43035
I20260928 10:57:45.437765 4844 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:45.438676 4844 heartbeater.cc:511] Master 127.3.252.126:43035 requested a full tablet report, sending...
I20260928 10:57:45.440822 4655 ts_manager.cc:194] Registered new tserver with Master: 19b6c3a6dd2e409abac9c5bf6175d62b (127.3.252.65:38193)
I20260928 10:57:45.442536 4655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:37881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:57:45.459465 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43035
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:45.778939 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:45.779204 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:45.779332 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:45.786614 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:45.786731 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:45.786885 4850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:57:45.796386 4850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:45.797634 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:45.799018 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:45.811661 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:45.811702 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:45.811633 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:45.812438 4850 server_base.cc:1034] running on GCE node
I20260928 10:57:45.813051 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:45.813997 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:45.815208 4850 hybrid_clock.cc:648] HybridClock initialized: now 1790593065815157 us; error 42 us; skew 500 ppm
I20260928 10:57:45.815513 4850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:45 dist-test-slave-djjw krb5kdc[4610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:45.818512 4850 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:57:45.819849 4850 webserver.cc:460] Webserver started at http://127.3.252.66:42089/ using document root <none> and password file <none>
I20260928 10:57:45.820559 4850 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:45.820657 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:45.820950 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:45.823797 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "e353a8216c1646fe9248bbe4fbcdfa00"
format_stamp: "Formatted at 2026-09-28 10:57:45 on dist-test-slave-djjw"
server_key: "4b734753a933fb3175a9d9eeb3764f25"
server_key_iv: "e543c53596fed172994d4dd7f5f854a8"
server_key_version: "encryptionkey@0"
I20260928 10:57:45.824560 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "e353a8216c1646fe9248bbe4fbcdfa00"
format_stamp: "Formatted at 2026-09-28 10:57:45 on dist-test-slave-djjw"
server_key: "4b734753a933fb3175a9d9eeb3764f25"
server_key_iv: "e543c53596fed172994d4dd7f5f854a8"
server_key_version: "encryptionkey@0"
I20260928 10:57:45.829674 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 10:57:45.833086 4865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:45.834513 4850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:57:45.834664 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "e353a8216c1646fe9248bbe4fbcdfa00"
format_stamp: "Formatted at 2026-09-28 10:57:45 on dist-test-slave-djjw"
server_key: "4b734753a933fb3175a9d9eeb3764f25"
server_key_iv: "e543c53596fed172994d4dd7f5f854a8"
server_key_version: "encryptionkey@0"
I20260928 10:57:45.834839 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:45.853655 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:45.857932 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:45.858217 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:45.859128 4850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:57:45.860381 4850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:45.860455 4850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:45.860539 4850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:45.860581 4850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:45.889652 4850 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:45461
I20260928 10:57:45.889695 4977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:45461 every 8 connection(s)
I20260928 10:57:45.890909 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:57:45.897710 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4850
I20260928 10:57:45.897913 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:57:45.898332 4081 external_mini_cluster.cc:1428] Setting key 61596d798319d11b5f83f3c4995c650f
Sep 28 10:57:45 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:45.910861 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:45.893397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:33511 (local address 127.3.252.126:43035)
0928 10:57:45.893572 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:45.893577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:45.894767 (+ 1190us) server_negotiation.cc:407] Connection header received
0928 10:57:45.896551 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:45.896556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:45.896652 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:45.896735 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:45.898282 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.899033 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.900991 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.901595 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.901723 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:45.905258 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:45.905263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:45.905266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:45.905300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:45.907615 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:45.908424 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:45.908428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:45.908430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:45.908549 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:45.908952 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:45.908955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:45.908957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:45.909231 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:45.909370 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:45.910444 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0928 10:57:45.910621 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:57:45.912385 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:45.893911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:43035 (local address 127.3.252.66:33511)
0928 10:57:45.894606 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:45.894652 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:45.896282 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:45.896979 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:45.896988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:45.897486 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:45.898123 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:45.898143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.899197 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.899201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:45.900845 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:45.900857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:45.901785 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:45.901789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:45.902048 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:45.902719 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:45.902747 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:45.905030 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:45.907747 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:45.907757 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:45.907776 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:45.908297 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:45.908649 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:45.908654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:45.908658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:45.908836 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:45.909544 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:45.909553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:45.910028 (+ 475us) client_negotiation.cc:769] Sending connection context
0928 10:57:45.910355 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 10:57:45.910939 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":242,"threads_started":1}
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 10:57:45.913867 4978 heartbeater.cc:347] Connected to a master server at 127.3.252.126:43035
I20260928 10:57:45.914179 4978 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:45.914933 4978 heartbeater.cc:511] Master 127.3.252.126:43035 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
I20260928 10:57:45.916419 4655 ts_manager.cc:194] Registered new tserver with Master: e353a8216c1646fe9248bbe4fbcdfa00 (127.3.252.66:45461)
I20260928 10:57:45.917181 4655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:33511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:57:45.936929 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43035
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:46.252077 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:46.252373 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:46.252485 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:46.260146 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:46.260267 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:46.260430 4983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:57:46.270184 4983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:46.271569 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:46.272845 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:46.286363 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:46.286391 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:46.286617 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:46.287798 4983 server_base.cc:1034] running on GCE node
I20260928 10:57:46.288434 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:46.289390 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:46.290599 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790593066290548 us; error 45 us; skew 500 ppm
I20260928 10:57:46.290923 4983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:46 dist-test-slave-djjw krb5kdc[4610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:46.294030 4983 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:57:46.295555 4983 webserver.cc:460] Webserver started at http://127.3.252.67:40439/ using document root <none> and password file <none>
I20260928 10:57:46.296216 4983 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:46.296314 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:46.296586 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:46.299526 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "22690d5498f749c5991cdd2ce41c0f2c"
format_stamp: "Formatted at 2026-09-28 10:57:46 on dist-test-slave-djjw"
server_key: "782ac2ba2044c80d655ba254fada8242"
server_key_iv: "a5a38cdcbbad8560e86ae37cbc6df8b7"
server_key_version: "encryptionkey@0"
I20260928 10:57:46.300277 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "22690d5498f749c5991cdd2ce41c0f2c"
format_stamp: "Formatted at 2026-09-28 10:57:46 on dist-test-slave-djjw"
server_key: "782ac2ba2044c80d655ba254fada8242"
server_key_iv: "a5a38cdcbbad8560e86ae37cbc6df8b7"
server_key_version: "encryptionkey@0"
I20260928 10:57:46.305554 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:57:46.309183 4998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:46.310621 4983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:57:46.310758 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "22690d5498f749c5991cdd2ce41c0f2c"
format_stamp: "Formatted at 2026-09-28 10:57:46 on dist-test-slave-djjw"
server_key: "782ac2ba2044c80d655ba254fada8242"
server_key_iv: "a5a38cdcbbad8560e86ae37cbc6df8b7"
server_key_version: "encryptionkey@0"
I20260928 10:57:46.310933 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:46.357113 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:46.361660 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:46.361967 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:46.363054 4983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:57:46.364409 4983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:46.364490 4983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:46.364580 4983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:46.364625 4983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:46.392383 4983 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:34267
I20260928 10:57:46.392482 5110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:34267 every 8 connection(s)
I20260928 10:57:46.393680 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:57:46.403285 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 4983
I20260928 10:57:46.403452 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:57:46.403847 4081 external_mini_cluster.cc:1428] Setting key 5200e8900a6ee2274f71887ed0f0a868
Sep 28 10:57:46 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:46.413667 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.396405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:55999 (local address 127.3.252.126:43035)
0928 10:57:46.396630 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.396635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.397748 (+ 1113us) server_negotiation.cc:407] Connection header received
0928 10:57:46.399657 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.399662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.399761 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.399853 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:46.401421 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.402229 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.403835 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.404261 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.404384 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.408071 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:46.408076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:46.408079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:46.408112 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:46.410536 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.411264 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.411279 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.411282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.411411 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.411867 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.411871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.411873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.412158 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:46.412273 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.413105 (+ 832us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.413332 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 10:57:46.414803 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.396867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:43035 (local address 127.3.252.67:55999)
0928 10:57:46.397585 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:46.397628 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:46.399362 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:46.400112 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:46.400122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:46.400640 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:46.401253 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.401270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.402354 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.402357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.403687 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.403698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.404492 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.404499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.404742 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.405411 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:46.405434 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:46.407875 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:46.410682 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:46.410690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:46.410704 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:46.411128 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:46.411514 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:46.411518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:46.411520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:46.411702 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:46.412269 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:46.412276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:46.412634 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 10:57:46.412963 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 10:57:46.413522 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
I20260928 10:57:46.416360 5111 heartbeater.cc:347] Connected to a master server at 127.3.252.126:43035
I20260928 10:57:46.416704 5111 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:46.417553 5111 heartbeater.cc:511] Master 127.3.252.126:43035 requested a full tablet report, sending...
I20260928 10:57:46.419117 4655 ts_manager.cc:194] Registered new tserver with Master: 22690d5498f749c5991cdd2ce41c0f2c (127.3.252.67:34267)
I20260928 10:57:46.419875 4655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:55999
I20260928 10:57:46.431433 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:57:46.445494 4844 heartbeater.cc:503] Master 127.3.252.126:43035 was elected leader, sending a full tablet report...
I20260928 10:57:46.450376 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.437548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.3.252.126:43035)
0928 10:57:46.437721 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.437728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.438163 (+ 435us) server_negotiation.cc:407] Connection header received
0928 10:57:46.438332 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.438336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.438416 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.438504 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:46.439945 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.440669 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.441978 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.442373 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.442482 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.445151 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:46.445157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:46.445160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:46.445216 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:46.447867 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.448475 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.448481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.448485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.448609 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.448962 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.448967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.448971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.449273 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:46.449507 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.449846 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.450073 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:57:46.456624 4655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:57:46.463371 4655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:46.502605 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.475936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:34267 (local address 127.0.0.1:43086)
0928 10:57:46.492911 (+ 16975us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:46.492947 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:46.493134 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:46.494019 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:46.494028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:46.494062 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:46.494490 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.494503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.496910 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.496917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.499490 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.499512 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.501542 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.501549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.501782 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.501869 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 10:57:46.501999 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 10:57:46.502204 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16709,"threads_started":1}
I20260928 10:57:46.504176 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.476007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.3.252.67:34267)
0928 10:57:46.492456 (+ 16449us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.492468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.493336 (+ 868us) server_negotiation.cc:407] Connection header received
0928 10:57:46.493448 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.493457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.493740 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.494698 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:46.495003 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.496735 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.499730 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.501299 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.503259 (+ 1960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.503536 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.503609 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.503804 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16241,"thread_start_us":16089,"threads_started":1}
I20260928 10:57:46.510434 5046 tablet_service.cc:1467] Processing CreateTablet for tablet b967d1e12f91424387dc5a85b07c3ede (DEFAULT_TABLE table=test-table [id=d44ffa80dbb1432cb42c8201f216a47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:46.511848 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b967d1e12f91424387dc5a85b07c3ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:46.513547 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.475967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38193 (local address 127.0.0.1:56468)
0928 10:57:46.502972 (+ 27005us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:46.502993 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:46.503100 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:46.506848 (+ 3748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:46.506853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:46.506886 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:46.507069 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.507078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.509080 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.509085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.510932 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.510951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.512705 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.512711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.512898 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.512932 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:57:46.513035 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 10:57:46.513170 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26893,"mutex_wait_us":4,"thread_start_us":16536,"threads_started":1}
I20260928 10:57:46.513584 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.476780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.3.252.65:38193)
0928 10:57:46.506165 (+ 29385us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.506174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.506203 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:57:46.506302 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.506311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.506611 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.507234 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:46.507449 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.508925 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.511105 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.512535 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.512669 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.512884 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.513108 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.513273 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29220,"thread_start_us":28937,"threads_started":1}
I20260928 10:57:46.514112 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.477155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.3.252.66:45461)
0928 10:57:46.505552 (+ 28397us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.505562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.506984 (+ 1422us) server_negotiation.cc:407] Connection header received
0928 10:57:46.507082 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.507087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.507246 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.507413 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:46.508196 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.509431 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.511209 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.512536 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.512669 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.512859 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.513637 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.513795 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28216,"thread_start_us":28103,"threads_started":1}
I20260928 10:57:46.514700 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.476307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:45461 (local address 127.0.0.1:56874)
0928 10:57:46.506415 (+ 30108us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:46.506435 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:46.506571 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:46.507430 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:46.507435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:46.507456 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:46.507780 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.507792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.509549 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.509554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.511036 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.511051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.513024 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.513030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.513199 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.513231 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:57:46.513315 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:57:46.513448 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29966,"mutex_wait_us":44,"thread_start_us":30476,"threads_started":1}
I20260928 10:57:46.517443 4779 tablet_service.cc:1467] Processing CreateTablet for tablet b967d1e12f91424387dc5a85b07c3ede (DEFAULT_TABLE table=test-table [id=d44ffa80dbb1432cb42c8201f216a47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:46.518297 4913 tablet_service.cc:1467] Processing CreateTablet for tablet b967d1e12f91424387dc5a85b07c3ede (DEFAULT_TABLE table=test-table [id=d44ffa80dbb1432cb42c8201f216a47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:46.518975 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b967d1e12f91424387dc5a85b07c3ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:46.519569 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b967d1e12f91424387dc5a85b07c3ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:46.524109 5129 tablet_bootstrap.cc:492] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c: Bootstrap starting.
I20260928 10:57:46.527101 5129 tablet_bootstrap.cc:654] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:46.528364 5129 log.cc:824] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:46.530923 5129 tablet_bootstrap.cc:492] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c: No bootstrap required, opened a new log
I20260928 10:57:46.531172 5129 ts_tablet_manager.cc:1397] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 10:57:46.534094 5131 tablet_bootstrap.cc:492] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b: Bootstrap starting.
I20260928 10:57:46.535684 5132 tablet_bootstrap.cc:492] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00: Bootstrap starting.
I20260928 10:57:46.537846 5131 tablet_bootstrap.cc:654] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:46.538947 5132 tablet_bootstrap.cc:654] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:46.539108 5131 log.cc:824] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:46.539911 5132 log.cc:824] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:46.540889 5129 raft_consensus.cc:348] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.541472 5129 raft_consensus.cc:374] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:46.541579 5129 raft_consensus.cc:729] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22690d5498f749c5991cdd2ce41c0f2c, State: Initialized, Role: FOLLOWER
I20260928 10:57:46.542448 5129 consensus_queue.cc:260] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.544486 5111 heartbeater.cc:503] Master 127.3.252.126:43035 was elected leader, sending a full tablet report...
I20260928 10:57:46.545118 5129 ts_tablet_manager.cc:1428] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 10:57:46.547794 5132 tablet_bootstrap.cc:492] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00: No bootstrap required, opened a new log
I20260928 10:57:46.548041 5132 ts_tablet_manager.cc:1397] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260928 10:57:46.548065 5131 tablet_bootstrap.cc:492] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b: No bootstrap required, opened a new log
I20260928 10:57:46.548312 5131 ts_tablet_manager.cc:1397] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260928 10:57:46.555799 5132 raft_consensus.cc:348] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.556272 5132 raft_consensus.cc:374] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:46.556380 5132 raft_consensus.cc:729] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e353a8216c1646fe9248bbe4fbcdfa00, State: Initialized, Role: FOLLOWER
I20260928 10:57:46.555794 5131 raft_consensus.cc:348] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.556756 5131 raft_consensus.cc:374] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:46.556860 5131 raft_consensus.cc:729] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b6c3a6dd2e409abac9c5bf6175d62b, State: Initialized, Role: FOLLOWER
I20260928 10:57:46.557070 5132 consensus_queue.cc:260] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.557413 5131 consensus_queue.cc:260] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.558274 4978 heartbeater.cc:503] Master 127.3.252.126:43035 was elected leader, sending a full tablet report...
I20260928 10:57:46.558935 5132 ts_tablet_manager.cc:1428] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 10:57:46.559062 5131 ts_tablet_manager.cc:1428] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260928 10:57:46.642860 4979 tablet.cc:2369] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:57:46.645313 5112 tablet.cc:2369] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:57:46.666836 4845 tablet.cc:2369] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:46.778079 5135 raft_consensus.cc:482] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:46.778371 5135 raft_consensus.cc:504] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.780067 5135 leader_election.cc:288] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19b6c3a6dd2e409abac9c5bf6175d62b (127.3.252.65:38193), e353a8216c1646fe9248bbe4fbcdfa00 (127.3.252.66:45461)
I20260928 10:57:46.785818 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.780617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:49711 (local address 127.3.252.65:38193)
0928 10:57:46.780870 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.780876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.780906 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:57:46.780977 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.780982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.781049 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.781124 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:46.781506 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.782277 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.783801 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.784973 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.785115 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.785188 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.785443 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.785584 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:57:46.785820 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.780391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38193 (local address 127.3.252.67:49711)
0928 10:57:46.780722 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:46.780743 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:46.780864 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:46.781167 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:46.781171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:46.781196 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:46.781369 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.781377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.782414 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.782417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.783648 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.783659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.785120 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.785125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.785294 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.785318 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:57:46.785395 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 10:57:46.785523 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260928 10:57:46.786350 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.780614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:44511 (local address 127.3.252.66:45461)
0928 10:57:46.780805 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.780812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.781389 (+ 577us) server_negotiation.cc:407] Connection header received
0928 10:57:46.781550 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.781554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.781608 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.781677 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:46.782168 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.782890 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.784498 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.785402 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.785533 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.785601 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.786038 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.786152 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:57:46.787026 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.780517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:45461 (local address 127.3.252.67:44511)
0928 10:57:46.781279 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:46.781296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:46.781418 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:46.781701 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:46.781705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:46.781728 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:46.782028 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.782036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.783039 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.783042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.784349 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:46.784361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.785586 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.785592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:46.785786 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.785858 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 10:57:46.786533 (+ 675us) client_negotiation.cc:240] Negotiation successful
0928 10:57:46.786661 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":445,"threads_started":1}
I20260928 10:57:46.786969 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b967d1e12f91424387dc5a85b07c3ede" candidate_uuid: "22690d5498f749c5991cdd2ce41c0f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" is_pre_election: true
I20260928 10:57:46.787534 4799 raft_consensus.cc:2454] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22690d5498f749c5991cdd2ce41c0f2c in term 0.
I20260928 10:57:46.787998 4933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b967d1e12f91424387dc5a85b07c3ede" candidate_uuid: "22690d5498f749c5991cdd2ce41c0f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" is_pre_election: true
I20260928 10:57:46.788161 4999 leader_election.cc:302] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b6c3a6dd2e409abac9c5bf6175d62b, 22690d5498f749c5991cdd2ce41c0f2c; no voters:
I20260928 10:57:46.788481 4933 raft_consensus.cc:2454] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22690d5498f749c5991cdd2ce41c0f2c in term 0.
I20260928 10:57:46.788525 5135 raft_consensus.cc:2781] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:46.788631 5135 raft_consensus.cc:482] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:46.788717 5135 raft_consensus.cc:3037] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:46.790962 5135 raft_consensus.cc:504] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.791545 5135 leader_election.cc:288] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [CANDIDATE]: Term 1 election: Requested vote from peers 19b6c3a6dd2e409abac9c5bf6175d62b (127.3.252.65:38193), e353a8216c1646fe9248bbe4fbcdfa00 (127.3.252.66:45461)
I20260928 10:57:46.791983 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b967d1e12f91424387dc5a85b07c3ede" candidate_uuid: "22690d5498f749c5991cdd2ce41c0f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b"
I20260928 10:57:46.792024 4933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b967d1e12f91424387dc5a85b07c3ede" candidate_uuid: "22690d5498f749c5991cdd2ce41c0f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e353a8216c1646fe9248bbe4fbcdfa00"
I20260928 10:57:46.792160 4799 raft_consensus.cc:3037] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:46.792192 4933 raft_consensus.cc:3037] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:46.794304 4933 raft_consensus.cc:2454] T b967d1e12f91424387dc5a85b07c3ede P e353a8216c1646fe9248bbe4fbcdfa00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22690d5498f749c5991cdd2ce41c0f2c in term 1.
I20260928 10:57:46.794310 4799 raft_consensus.cc:2454] T b967d1e12f91424387dc5a85b07c3ede P 19b6c3a6dd2e409abac9c5bf6175d62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22690d5498f749c5991cdd2ce41c0f2c in term 1.
I20260928 10:57:46.794690 5000 leader_election.cc:302] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22690d5498f749c5991cdd2ce41c0f2c, e353a8216c1646fe9248bbe4fbcdfa00; no voters:
I20260928 10:57:46.794965 5135 raft_consensus.cc:2781] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:46.795373 5135 raft_consensus.cc:686] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [term 1 LEADER]: Becoming Leader. State: Replica: 22690d5498f749c5991cdd2ce41c0f2c, State: Running, Role: LEADER
I20260928 10:57:46.795802 5135 consensus_queue.cc:237] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } }
I20260928 10:57:46.800489 4654 catalog_manager.cc:5495] T b967d1e12f91424387dc5a85b07c3ede P 22690d5498f749c5991cdd2ce41c0f2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 22690d5498f749c5991cdd2ce41c0f2c (127.3.252.67). New cstate: current_term: 1 leader_uuid: "22690d5498f749c5991cdd2ce41c0f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22690d5498f749c5991cdd2ce41c0f2c" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19b6c3a6dd2e409abac9c5bf6175d62b" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e353a8216c1646fe9248bbe4fbcdfa00" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 45461 } health_report { overall_health: UNKNOWN } } }
Sep 28 10:57:46 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:57:46.862433 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.844547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.3.252.65:38193)
0928 10:57:46.844701 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.844705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.845185 (+ 480us) server_negotiation.cc:407] Connection header received
0928 10:57:46.845354 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.845358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.845448 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.845535 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:46.847431 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.848151 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.849404 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.849855 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.849972 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.857092 (+ 7120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:46.857115 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:46.857135 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:46.857175 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:46.859910 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.860571 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.860578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.860581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.860722 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.861096 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.861099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.861101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.861452 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:46.861665 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.862001 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.862164 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 10:57:46 dist-test-slave-djjw krb5kdc[4610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:57:46 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:57:46.906211 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.891963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.3.252.65:38193)
0928 10:57:46.892151 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.892157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.892529 (+ 372us) server_negotiation.cc:407] Connection header received
0928 10:57:46.892674 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.892680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.892760 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.892868 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:46.894269 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.895359 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.896721 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.897361 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.897483 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.901174 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:46.901180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:46.901185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:46.901225 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:46.904138 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.904704 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.904707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.904709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.904828 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.905127 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.905130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.905132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.905385 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:46.905586 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.905752 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.905970 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 10:57:46 dist-test-slave-djjw krb5kdc[4610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:57:46 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:57:46.947891 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.934263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.3.252.65:38193)
0928 10:57:46.934452 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.934459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.934760 (+ 301us) server_negotiation.cc:407] Connection header received
0928 10:57:46.934936 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.934941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.935031 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.935117 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:46.936467 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.937161 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.938489 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.938925 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.939051 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.942803 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:46.942809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:46.942812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:46.942850 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:46.945345 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.946030 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.946036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.946038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.946159 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.946513 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.946516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.946518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.946812 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:46.947055 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.947336 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.947565 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 10:57:46.949524 4759 tablet_service.cc:2908] Rejecting scan request for tablet b967d1e12f91424387dc5a85b07c3ede: Uninitialized: clean time has not yet been initialized
I20260928 10:57:46.952499 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4716
I20260928 10:57:46.965086 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:46.953336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.3.252.65:38193)
0928 10:57:46.953514 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:46.953520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:46.953793 (+ 273us) server_negotiation.cc:407] Connection header received
0928 10:57:46.954002 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:46.954006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:46.954077 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:46.954156 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:46.955274 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.956090 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.957300 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:46.957689 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:46.957805 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:46.959941 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:46.959946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:46.959949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:46.959980 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:46.962373 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.962984 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.962989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.962991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.963108 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:46.963487 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:46.963490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:46.963492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:46.964187 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:46.964421 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:46.964683 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 10:57:46.964857 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 10:57:46.965754 4839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:47.056104 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4716
I20260928 10:57:47.069983 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4850
Sep 28 10:57:47 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:57:47.083434 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:47.070691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.3.252.66:45461)
0928 10:57:47.070837 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:47.070842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:47.070875 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:57:47.071030 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:47.071035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:47.071106 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:47.071188 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:47.072265 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:47.072980 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:47.074347 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:47.074736 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:47.074825 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:47.078560 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:47.078574 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:47.078591 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:47.078623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:47.081026 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:47.081669 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:47.081679 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:47.081682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:47.081809 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:47.082185 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:47.082188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:47.082190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:47.082537 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:47.082773 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:47.083025 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 10:57:47.083184 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 10:57:47.084054 4973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:47.173249 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4850
I20260928 10:57:47.188210 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4983
Sep 28 10:57:47 dist-test-slave-djjw krb5kdc[4610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:57:47.202610 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:47.188924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.3.252.67:34267)
0928 10:57:47.189412 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:47.189417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:47.189433 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:57:47.189486 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:47.189489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:47.189547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:47.189638 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:47.190901 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:47.191592 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:47.192826 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:47.193223 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:47.193340 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:47.197240 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:47.197256 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:47.197272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:47.197316 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:47.199933 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:47.200621 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:47.200630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:47.200634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:47.200775 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:47.201130 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:47.201134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:47.201136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:47.201469 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:47.201726 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:47.202049 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 10:57:47.202254 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":228,"threads_started":1}
I20260928 10:57:47.203217 5106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:47.291589 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4983
I20260928 10:57:47.304745 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 4623
I20260928 10:57:47.305838 4685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:47.410732 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 4623
2026-09-28T10:57:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4164 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:47 dist-test-slave-djjw krb5kdc[5160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:47 dist-test-slave-djjw krb5kdc[5160](info): set up 2 sockets
Sep 28 10:57:47 dist-test-slave-djjw krb5kdc[5160](info): commencing operation
krb5kdc: starting...
W20260928 10:57:48.414158 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:57:48 dist-test-slave-djjw krb5kdc[5160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:57:48.589753 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36399
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:48.904381 5173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:48.904673 5173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:48.904806 5173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:48.912048 5173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:48.912154 5173 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:57:48.912194 5173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:48.912271 5173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:48.912338 5173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:48.912418 5173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:48.922416 5173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36399
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:48.923714 5173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:48.924999 5173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:48.936425 5178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:48.936548 5181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:48.936426 5179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:48.937233 5173 server_base.cc:1034] running on GCE node
I20260928 10:57:48.938103 5173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:48.939523 5173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:48.940755 5173 hybrid_clock.cc:648] HybridClock initialized: now 1790593068940688 us; error 38 us; skew 500 ppm
I20260928 10:57:48.941139 5173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:48 dist-test-slave-djjw krb5kdc[5160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593068, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:48.944257 5173 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:57:48.945783 5173 webserver.cc:460] Webserver started at http://127.3.252.126:34695/ using document root <none> and password file <none>
I20260928 10:57:48.946663 5173 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:48.946772 5173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:48.947118 5173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:48.950357 5173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "b6364ec1e19a4350ac2c4544dfb65871"
format_stamp: "Formatted at 2026-09-28 10:57:48 on dist-test-slave-djjw"
server_key: "3b18d71a47ed23a489c794bcea20136d"
server_key_iv: "09b13746eba0f091c5a0d17807cc166a"
server_key_version: "encryptionkey@0"
I20260928 10:57:48.951150 5173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "b6364ec1e19a4350ac2c4544dfb65871"
format_stamp: "Formatted at 2026-09-28 10:57:48 on dist-test-slave-djjw"
server_key: "3b18d71a47ed23a489c794bcea20136d"
server_key_iv: "09b13746eba0f091c5a0d17807cc166a"
server_key_version: "encryptionkey@0"
I20260928 10:57:48.956631 5173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:57:48.960542 5188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:48.962196 5173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 10:57:48.962389 5173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "b6364ec1e19a4350ac2c4544dfb65871"
format_stamp: "Formatted at 2026-09-28 10:57:48 on dist-test-slave-djjw"
server_key: "3b18d71a47ed23a489c794bcea20136d"
server_key_iv: "09b13746eba0f091c5a0d17807cc166a"
server_key_version: "encryptionkey@0"
I20260928 10:57:48.962612 5173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:48.977993 5173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:48.982429 5173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:48.982688 5173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:49.002731 5173 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:36399
I20260928 10:57:49.002741 5249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:36399 every 8 connection(s)
I20260928 10:57:49.004168 5173 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:49.006064 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 5173
I20260928 10:57:49.006292 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:57:49.007093 4081 external_mini_cluster.cc:1428] Setting key 1132fd306dc7098ea3edbe96c00a3947
I20260928 10:57:49.009187 5250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:57:49 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:49.022719 5250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871: Bootstrap starting.
I20260928 10:57:49.026324 5250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:49.027525 5250 log.cc:824] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:49.031131 5250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871: No bootstrap required, opened a new log
I20260928 10:57:49.032061 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:49.009233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.3.252.126:36399)
0928 10:57:49.010002 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:49.010015 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:49.010057 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:57:49.011647 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:49.011670 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:49.012094 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:49.012441 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:49.014209 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.015686 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.017415 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.018167 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.018323 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:49.023437 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:49.023446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:49.023465 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:49.023511 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:49.027115 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:49.028100 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:49.028111 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:49.028122 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:49.028393 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:49.028945 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:49.028951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:49.028954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:49.029674 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:49.029978 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:49.030524 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 10:57:49.030947 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":324,"threads_started":1}
I20260928 10:57:49.039306 5250 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6364ec1e19a4350ac2c4544dfb65871" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36399 } }
I20260928 10:57:49.039762 5250 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:49.039867 5250 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6364ec1e19a4350ac2c4544dfb65871, State: Initialized, Role: FOLLOWER
I20260928 10:57:49.040503 5250 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6364ec1e19a4350ac2c4544dfb65871" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36399 } }
I20260928 10:57:49.040751 5250 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:49.040896 5250 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:49.041083 5250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:49.043426 5250 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6364ec1e19a4350ac2c4544dfb65871" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36399 } }
I20260928 10:57:49.043956 5250 leader_election.cc:302] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6364ec1e19a4350ac2c4544dfb65871; no voters:
I20260928 10:57:49.044551 5250 leader_election.cc:288] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:49.044678 5255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:49.045328 5255 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [term 1 LEADER]: Becoming Leader. State: Replica: b6364ec1e19a4350ac2c4544dfb65871, State: Running, Role: LEADER
I20260928 10:57:49.045956 5255 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6364ec1e19a4350ac2c4544dfb65871" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36399 } }
I20260928 10:57:49.046658 5250 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:49.048619 5256 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6364ec1e19a4350ac2c4544dfb65871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6364ec1e19a4350ac2c4544dfb65871" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36399 } } }
I20260928 10:57:49.049012 5256 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:49.049726 5257 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6364ec1e19a4350ac2c4544dfb65871. Latest consensus state: current_term: 1 leader_uuid: "b6364ec1e19a4350ac2c4544dfb65871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6364ec1e19a4350ac2c4544dfb65871" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36399 } } }
I20260928 10:57:49.049955 5260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:49.050071 5257 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:49.056432 5260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:49.066627 5260 catalog_manager.cc:1347] Generated new cluster ID: 819fff3d6186445c8fbe4b2aec89bdf7
I20260928 10:57:49.066861 5260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:49.079156 5260 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:49.081064 5260 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:49.093567 5260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871: Generated new TSK 0
I20260928 10:57:49.094811 5260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:57:49.138007 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36399
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:49.459583 5276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:49.459897 5276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:49.460037 5276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:49.467650 5276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:49.467780 5276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:49.467911 5276 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:57:49.468022 5276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:57:49.477861 5276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:36399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:49.479153 5276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:49.480474 5276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:49.493520 5281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:49.493610 5282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:49.493637 5284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:49.494158 5276 server_base.cc:1034] running on GCE node
I20260928 10:57:49.494843 5276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:49.495771 5276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:49.497021 5276 hybrid_clock.cc:648] HybridClock initialized: now 1790593069496951 us; error 48 us; skew 500 ppm
I20260928 10:57:49.497349 5276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:49 dist-test-slave-djjw krb5kdc[5160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593069, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:49.500479 5276 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:57:49.501986 5276 webserver.cc:460] Webserver started at http://127.3.252.65:34185/ using document root <none> and password file <none>
I20260928 10:57:49.502696 5276 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:49.502810 5276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:49.503124 5276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:49.506141 5276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "323f35481c86457bb43f2dee5824fc7d"
format_stamp: "Formatted at 2026-09-28 10:57:49 on dist-test-slave-djjw"
server_key: "b980be493ca15bf85ca8dabc76e58e9d"
server_key_iv: "141f9da34709c0772b0145ab92918c2a"
server_key_version: "encryptionkey@0"
I20260928 10:57:49.506896 5276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "323f35481c86457bb43f2dee5824fc7d"
format_stamp: "Formatted at 2026-09-28 10:57:49 on dist-test-slave-djjw"
server_key: "b980be493ca15bf85ca8dabc76e58e9d"
server_key_iv: "141f9da34709c0772b0145ab92918c2a"
server_key_version: "encryptionkey@0"
I20260928 10:57:49.511970 5276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:57:49.515679 5291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:49.517148 5276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:57:49.517313 5276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "323f35481c86457bb43f2dee5824fc7d"
format_stamp: "Formatted at 2026-09-28 10:57:49 on dist-test-slave-djjw"
server_key: "b980be493ca15bf85ca8dabc76e58e9d"
server_key_iv: "141f9da34709c0772b0145ab92918c2a"
server_key_version: "encryptionkey@0"
I20260928 10:57:49.517517 5276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:49.536247 5276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:49.541440 5276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:49.541718 5276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:49.544498 5276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:49.544585 5276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:49.544723 5276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:49.544780 5276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:57:49 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593069, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:49.567446 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:49.545045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:34607 (local address 127.3.252.126:36399)
0928 10:57:49.545636 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:49.545643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:49.546908 (+ 1265us) server_negotiation.cc:407] Connection header received
0928 10:57:49.549264 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:49.549269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:49.549368 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:49.549491 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:49.551717 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.552876 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.555943 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.556446 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.556583 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:49.560903 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:49.560912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:49.560918 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:49.560967 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:49.564201 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:49.564999 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:49.565004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:49.565006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:49.565125 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:49.565563 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:49.565566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:49.565569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:49.565911 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:49.566055 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:49.566822 (+ 767us) server_negotiation.cc:299] Negotiation successful
0928 10:57:49.567086 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":454,"threads_started":1}
I20260928 10:57:49.568529 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:49.545684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36399 (local address 127.3.252.65:34607)
0928 10:57:49.546708 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:49.546762 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:49.548953 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:49.549773 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:49.549784 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:49.550490 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:49.551392 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:49.551421 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.553154 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.553160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:49.555758 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:49.555778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.556922 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.556930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:49.557281 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:49.557959 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:49.557981 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:49.560589 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:49.564363 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:49.564374 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:49.564393 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:49.564859 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:49.565231 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:49.565236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:49.565240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:49.565446 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:49.566030 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:49.566040 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:49.566449 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 10:57:49.566688 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 10:57:49.567240 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":360,"threads_started":1}
I20260928 10:57:49.590941 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:49.576480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36399 (local address 127.3.252.65:36945)
0928 10:57:49.577190 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:49.577213 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:49.577415 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:49.577750 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:49.577755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:49.578607 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:49.579164 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:49.579187 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.580417 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.580423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:49.582027 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:49.582045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.583049 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.583057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:49.583208 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:49.583830 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:49.583847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:49.584588 (+ 741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:49.587900 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:49.587904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:49.587907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:49.588358 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:49.588827 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:49.588832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:49.588836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:49.589031 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:49.589656 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:49.589659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:49.589903 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 10:57:49.590061 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 10:57:49.590407 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":345,"threads_started":1}
I20260928 10:57:49.591780 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:49.576483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:36945 (local address 127.3.252.126:36399)
0928 10:57:49.576667 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:49.576674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:49.577339 (+ 665us) server_negotiation.cc:407] Connection header received
0928 10:57:49.577543 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:49.577548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:49.577617 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:49.577703 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:49.579358 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.580233 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.582236 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:49.582849 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:49.582966 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:49.584784 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:49.584796 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:49.584799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:49.584832 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:49.587766 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:49.588483 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:49.588489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:49.588493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:49.588668 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:49.589153 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:49.589159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:49.589162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:49.589504 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:49.591252 (+ 1748us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:49.591376 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 10:57:49.591559 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:57:49.599086 5276 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:39911
I20260928 10:57:49.599128 5411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:39911 every 8 connection(s)
I20260928 10:57:49.600250 5276 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:57:49.602279 5412 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36399
I20260928 10:57:49.602484 5412 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:49.603116 5412 heartbeater.cc:511] Master 127.3.252.126:36399 requested a full tablet report, sending...
I20260928 10:57:49.604974 5205 ts_manager.cc:194] Registered new tserver with Master: 323f35481c86457bb43f2dee5824fc7d (127.3.252.65:39911)
I20260928 10:57:49.606505 5205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:34607
I20260928 10:57:49.606909 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 5276
I20260928 10:57:49.607128 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:57:49.607786 4081 external_mini_cluster.cc:1428] Setting key 93aa9463168b71d27682f0965ccfa4b7
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:57:49.646245 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36399
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:49.958622 5416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:49.958921 5416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:49.959038 5416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:49.966306 5416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:49.966434 5416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:49.966552 5416 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:57:49.966637 5416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:57:49.976625 5416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:36399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:49.977888 5416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:49.979144 5416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:49.991833 5421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:49.991777 5422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:49.991811 5424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:49.992595 5416 server_base.cc:1034] running on GCE node
I20260928 10:57:49.993193 5416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:49.994172 5416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:49.995409 5416 hybrid_clock.cc:648] HybridClock initialized: now 1790593069995341 us; error 36 us; skew 500 ppm
I20260928 10:57:49.995721 5416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:49 dist-test-slave-djjw krb5kdc[5160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593069, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:49.998836 5416 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:57:50.000321 5416 webserver.cc:460] Webserver started at http://127.3.252.66:38223/ using document root <none> and password file <none>
I20260928 10:57:50.001036 5416 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:50.001124 5416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:50.001387 5416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:50.004462 5416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "c0fd1a2ed2a0449eb637758b3672e7c7"
format_stamp: "Formatted at 2026-09-28 10:57:49 on dist-test-slave-djjw"
server_key: "3a9b074f6356cd57b02c0ea8497af238"
server_key_iv: "4bceb5932499195f4d20617104fb1741"
server_key_version: "encryptionkey@0"
I20260928 10:57:50.005313 5416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "c0fd1a2ed2a0449eb637758b3672e7c7"
format_stamp: "Formatted at 2026-09-28 10:57:49 on dist-test-slave-djjw"
server_key: "3a9b074f6356cd57b02c0ea8497af238"
server_key_iv: "4bceb5932499195f4d20617104fb1741"
server_key_version: "encryptionkey@0"
I20260928 10:57:50.010600 5416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 10:57:50.014322 5431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:50.015903 5416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:57:50.016098 5416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "c0fd1a2ed2a0449eb637758b3672e7c7"
format_stamp: "Formatted at 2026-09-28 10:57:49 on dist-test-slave-djjw"
server_key: "3a9b074f6356cd57b02c0ea8497af238"
server_key_iv: "4bceb5932499195f4d20617104fb1741"
server_key_version: "encryptionkey@0"
I20260928 10:57:50.016332 5416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:50.032723 5416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:50.036976 5416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:50.037206 5416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:50.039955 5416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:50.040048 5416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:50.040248 5416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:50.040309 5416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:57:50 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593069, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:50.064853 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.040512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:42113 (local address 127.3.252.126:36399)
0928 10:57:50.040671 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.040675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.042192 (+ 1517us) server_negotiation.cc:407] Connection header received
0928 10:57:50.045013 (+ 2821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.045020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.045141 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.045249 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:50.047703 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.048738 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.051760 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.052370 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.052509 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.058024 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:50.058033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:50.058038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:50.058085 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:50.060947 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.062021 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.062028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.062032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.062186 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.062705 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.062711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.062715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.063126 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:50.063300 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.064259 (+ 959us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.064514 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 10:57:50.066083 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.041110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36399 (local address 127.3.252.66:42113)
0928 10:57:50.041996 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.042050 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.044572 (+ 2522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.045695 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.045708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.046479 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:50.047490 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.047531 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.048948 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.048954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.051566 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.051590 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.052869 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.052878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.053248 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.054113 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:50.054152 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:50.057640 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:50.061102 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.061113 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.061133 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.061810 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.062324 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.062329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.062332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.062559 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.063277 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:50.063286 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:50.063792 (+ 506us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.064160 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.064721 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":317,"threads_started":1}
I20260928 10:57:50.086083 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.072230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:49145 (local address 127.3.252.126:36399)
0928 10:57:50.072421 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.072428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.072670 (+ 242us) server_negotiation.cc:407] Connection header received
0928 10:57:50.072855 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.072861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.072948 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.073034 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:50.074415 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.075424 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.077273 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.077919 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.078070 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.080217 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:50.080223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:50.080226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:50.080258 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:50.082917 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.083781 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.083787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.083791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.083949 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.084369 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.084374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.084378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.084869 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:50.085030 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.085498 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.085718 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:57:50.086246 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.071947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36399 (local address 127.3.252.66:49145)
0928 10:57:50.072574 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.072590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.072734 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.073121 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.073129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.073733 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:50.074211 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.074226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.075586 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.075590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.077136 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.077148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.078133 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.078138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.078253 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.078900 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:50.078921 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:50.080043 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:50.083082 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.083086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.083089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.083621 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.084059 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.084067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.084071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.084246 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.085000 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:50.085005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:50.085246 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.085379 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.085744 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":346,"threads_started":1}
I20260928 10:57:50.087594 5416 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:33359
I20260928 10:57:50.087643 5550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:33359 every 8 connection(s)
I20260928 10:57:50.088830 5416 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:57:50.090950 5551 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36399
I20260928 10:57:50.091173 5551 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:50.091797 5551 heartbeater.cc:511] Master 127.3.252.126:36399 requested a full tablet report, sending...
I20260928 10:57:50.092790 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 5416
I20260928 10:57:50.093024 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:57:50.093231 5205 ts_manager.cc:194] Registered new tserver with Master: c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66:33359)
I20260928 10:57:50.093477 4081 external_mini_cluster.cc:1428] Setting key 10b12d65497ce77d9a0624826350d812
I20260928 10:57:50.094254 5205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:42113
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:57:50.132731 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36399
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:50.463739 5555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:50.464036 5555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:50.464131 5555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:50.471606 5555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:50.471731 5555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:50.471846 5555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:57:50.471937 5555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:57:50.481696 5555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:36399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:50.483011 5555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:50.484287 5555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:50.496709 5561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:50.496671 5563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:50.497197 5560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:50.497434 5555 server_base.cc:1034] running on GCE node
I20260928 10:57:50.498086 5555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:50.498958 5555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:50.500164 5555 hybrid_clock.cc:648] HybridClock initialized: now 1790593070500101 us; error 35 us; skew 500 ppm
I20260928 10:57:50.500450 5555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:50 dist-test-slave-djjw krb5kdc[5160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593070, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:50.503571 5555 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:57:50.504940 5555 webserver.cc:460] Webserver started at http://127.3.252.67:34837/ using document root <none> and password file <none>
I20260928 10:57:50.505645 5555 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:50.505733 5555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:50.506055 5555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:50.508921 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0"
format_stamp: "Formatted at 2026-09-28 10:57:50 on dist-test-slave-djjw"
server_key: "e6fd085a1d07b6b4c325e1477cc84ba2"
server_key_iv: "f2a259f7c8e07a57ab5aafd7f822b329"
server_key_version: "encryptionkey@0"
I20260928 10:57:50.509716 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0"
format_stamp: "Formatted at 2026-09-28 10:57:50 on dist-test-slave-djjw"
server_key: "e6fd085a1d07b6b4c325e1477cc84ba2"
server_key_iv: "f2a259f7c8e07a57ab5aafd7f822b329"
server_key_version: "encryptionkey@0"
I20260928 10:57:50.515669 5555 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 10:57:50.520628 5570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:50.522305 5555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:57:50.522487 5555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0"
format_stamp: "Formatted at 2026-09-28 10:57:50 on dist-test-slave-djjw"
server_key: "e6fd085a1d07b6b4c325e1477cc84ba2"
server_key_iv: "f2a259f7c8e07a57ab5aafd7f822b329"
server_key_version: "encryptionkey@0"
I20260928 10:57:50.522720 5555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:50.542104 5555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:50.546482 5555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:50.546725 5555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:50.549470 5555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:50.549551 5555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:50.549682 5555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:50.549737 5555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:57:50 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593070, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:50.581748 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.551249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36399 (local address 127.3.252.67:43153)
0928 10:57:50.552591 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.552642 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.558868 (+ 6226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.559941 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.559953 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.560732 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:50.561615 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.561644 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.563290 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.563296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.565585 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.565602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.567001 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.567010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.567341 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.568233 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:50.568270 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:50.571761 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:50.576030 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.576041 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.576066 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.576803 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.577312 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.577318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.577322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.577625 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.578428 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:50.578437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:50.579047 (+ 610us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.579398 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.580018 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":717,"threads_started":1}
I20260928 10:57:50.582715 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.551848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:43153 (local address 127.3.252.126:36399)
0928 10:57:50.552060 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.552067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.552767 (+ 700us) server_negotiation.cc:407] Connection header received
0928 10:57:50.559269 (+ 6502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.559277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.559374 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.559490 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:50.561877 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.563095 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.565771 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.566541 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.566679 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.572121 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:50.572129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:50.572134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:50.572180 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:50.575830 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.576986 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.576993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.576997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.577178 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.577789 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.577795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.577798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.578196 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:50.578334 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.581671 (+ 3337us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.582297 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:57:50.609660 5412 heartbeater.cc:503] Master 127.3.252.126:36399 was elected leader, sending a full tablet report...
I20260928 10:57:50.611129 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.593724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36399 (local address 127.3.252.67:59321)
0928 10:57:50.595023 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.595046 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.595226 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.595680 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.595685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.596289 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:50.596677 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.596690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.598282 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.598288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.600237 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.600255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.601461 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.601469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.601675 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.602585 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:50.602613 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:50.603965 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:50.608124 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.608130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.608135 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.608693 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.609086 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:50.609091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:50.609095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:50.609247 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:50.609867 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:50.609872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:50.610090 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.610230 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.610606 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"thread_start_us":988,"threads_started":1}
I20260928 10:57:50.612192 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.593726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:59321 (local address 127.3.252.126:36399)
0928 10:57:50.593936 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.593943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.595174 (+ 1231us) server_negotiation.cc:407] Connection header received
0928 10:57:50.595348 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.595354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.595450 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.595556 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:50.596859 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.598097 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.600431 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.601035 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.601174 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.604186 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:50.604194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:50.604199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:50.604244 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:50.607972 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.608815 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.608821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.608825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.608981 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.609344 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.609349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.609353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.609703 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:50.611438 (+ 1735us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.611592 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.611844 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:57:50.619701 5555 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:46651
I20260928 10:57:50.619774 5689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:46651 every 8 connection(s)
I20260928 10:57:50.620980 5555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:57:50.623139 5690 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36399
I20260928 10:57:50.623371 5690 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:50.624202 5690 heartbeater.cc:511] Master 127.3.252.126:36399 requested a full tablet report, sending...
I20260928 10:57:50.624785 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 5555
I20260928 10:57:50.624955 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:57:50.625355 4081 external_mini_cluster.cc:1428] Setting key ccd72270372d9c9ee90fcb6d56e26188
I20260928 10:57:50.625865 5204 ts_manager.cc:194] Registered new tserver with Master: e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67:46651)
I20260928 10:57:50.627147 5204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:43153
I20260928 10:57:50.628295 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:57:50 dist-test-slave-djjw krb5kdc[5160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:57:50 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:50.680178 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.663916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.3.252.126:36399)
0928 10:57:50.664119 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.664125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.664268 (+ 143us) server_negotiation.cc:407] Connection header received
0928 10:57:50.664477 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.664482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.664556 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.664633 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:50.666490 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.667323 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.669234 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.669664 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.669769 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.674946 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:50.674952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:50.674955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:50.674995 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:50.677551 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.678276 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.678281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.678284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.678414 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:50.678791 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:50.678795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:50.678797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:50.679084 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:50.679312 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.679638 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.679872 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:57:50.684654 5205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:57:50.691772 5205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:50.728259 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.705753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.3.252.66:33359)
0928 10:57:50.715656 (+ 9903us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.715668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.717270 (+ 1602us) server_negotiation.cc:407] Connection header received
0928 10:57:50.718526 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.718534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.718790 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.718974 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:50.720025 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.721931 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.725007 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.726558 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.726684 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.726927 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.727782 (+ 855us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.727958 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9667,"thread_start_us":9924,"threads_started":1}
I20260928 10:57:50.729135 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.705318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:33359 (local address 127.0.0.1:47694)
0928 10:57:50.717158 (+ 11840us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.717180 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.718282 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.719265 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.719274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.719326 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:50.719774 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.719788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.722148 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.722154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.724813 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.724826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.727168 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.727179 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.727466 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.727536 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.728547 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.728759 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11722,"mutex_wait_us":5,"thread_start_us":11487,"threads_started":1}
I20260928 10:57:50.733481 5481 tablet_service.cc:1467] Processing CreateTablet for tablet 0b7a00b44e8b47e29f20c2c5940201df (DEFAULT_TABLE table=test-table [id=5fb015f3484b41db873b06f1305645a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:50.735045 5481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b7a00b44e8b47e29f20c2c5940201df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:50.735930 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.705632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.3.252.65:39911)
0928 10:57:50.716583 (+ 10951us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.716593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.716986 (+ 393us) server_negotiation.cc:407] Connection header received
0928 10:57:50.717471 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.717480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.717754 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.717990 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:50.727415 (+ 9425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.729498 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.732157 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.733859 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.733981 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.734296 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.735415 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.735615 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10787,"thread_start_us":10414,"threads_started":1}
I20260928 10:57:50.736999 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.704839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39911 (local address 127.0.0.1:39314)
0928 10:57:50.716803 (+ 11964us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.716854 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.717113 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.726733 (+ 9620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.726738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.726766 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:50.727202 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.727216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.729712 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.729718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.731952 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.731969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.734825 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.734833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.735058 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.735100 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.736367 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.736542 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11659,"thread_start_us":11499,"threads_started":1}
I20260928 10:57:50.741169 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.717801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:46651 (local address 127.0.0.1:43358)
0928 10:57:50.726231 (+ 8430us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.726258 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.726391 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.730747 (+ 4356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.730752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.730781 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:50.731153 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.731165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.734686 (+ 3521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.734691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.736672 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.736687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.740347 (+ 3660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.740354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.740532 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.740566 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.740656 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.740808 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8278,"mutex_wait_us":1,"thread_start_us":8643,"threads_started":1}
I20260928 10:57:50.741927 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.717964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.3.252.67:46651)
0928 10:57:50.729755 (+ 11791us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.729763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.729797 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:57:50.729917 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.729924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.730202 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.731386 (+ 1184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:50.731679 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.734509 (+ 2830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.736888 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.739856 (+ 2968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.739987 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.740211 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.741411 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.741576 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11615,"thread_start_us":12513,"threads_started":1}
I20260928 10:57:50.741366 5342 tablet_service.cc:1467] Processing CreateTablet for tablet 0b7a00b44e8b47e29f20c2c5940201df (DEFAULT_TABLE table=test-table [id=5fb015f3484b41db873b06f1305645a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:50.743157 5342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b7a00b44e8b47e29f20c2c5940201df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:50.746053 5620 tablet_service.cc:1467] Processing CreateTablet for tablet 0b7a00b44e8b47e29f20c2c5940201df (DEFAULT_TABLE table=test-table [id=5fb015f3484b41db873b06f1305645a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:50.747581 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b7a00b44e8b47e29f20c2c5940201df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:50.763324 5709 tablet_bootstrap.cc:492] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Bootstrap starting.
I20260928 10:57:50.765677 5710 tablet_bootstrap.cc:492] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Bootstrap starting.
I20260928 10:57:50.768414 5709 tablet_bootstrap.cc:654] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:50.769563 5710 tablet_bootstrap.cc:654] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:50.770947 5710 log.cc:824] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:50.771041 5709 log.cc:824] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:50.773114 5711 tablet_bootstrap.cc:492] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Bootstrap starting.
I20260928 10:57:50.777663 5711 tablet_bootstrap.cc:654] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:50.778960 5710 tablet_bootstrap.cc:492] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: No bootstrap required, opened a new log
I20260928 10:57:50.779030 5711 log.cc:824] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:50.779392 5710 ts_tablet_manager.cc:1397] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260928 10:57:50.788731 5711 tablet_bootstrap.cc:492] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: No bootstrap required, opened a new log
I20260928 10:57:50.789045 5711 ts_tablet_manager.cc:1397] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260928 10:57:50.790010 5709 tablet_bootstrap.cc:492] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: No bootstrap required, opened a new log
I20260928 10:57:50.790320 5709 ts_tablet_manager.cc:1397] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.015s
I20260928 10:57:50.792546 5710 raft_consensus.cc:348] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.793006 5710 raft_consensus.cc:374] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:50.793107 5710 raft_consensus.cc:729] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a73e7a7ac54f3a8f1fe4866c4b1ad0, State: Initialized, Role: FOLLOWER
I20260928 10:57:50.793759 5710 consensus_queue.cc:260] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.801456 5711 raft_consensus.cc:348] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.802067 5711 raft_consensus.cc:374] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:50.802160 5711 raft_consensus.cc:729] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323f35481c86457bb43f2dee5824fc7d, State: Initialized, Role: FOLLOWER
I20260928 10:57:50.802951 5711 consensus_queue.cc:260] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.803714 5709 raft_consensus.cc:348] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.804273 5709 raft_consensus.cc:374] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:50.804407 5709 raft_consensus.cc:729] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1a2ed2a0449eb637758b3672e7c7, State: Initialized, Role: FOLLOWER
I20260928 10:57:50.805207 5709 consensus_queue.cc:260] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.810537 5690 heartbeater.cc:503] Master 127.3.252.126:36399 was elected leader, sending a full tablet report...
I20260928 10:57:50.811082 5711 ts_tablet_manager.cc:1428] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20260928 10:57:50.811326 5710 ts_tablet_manager.cc:1428] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Time spent starting tablet: real 0.032s user 0.017s sys 0.012s
I20260928 10:57:50.813265 5551 heartbeater.cc:503] Master 127.3.252.126:36399 was elected leader, sending a full tablet report...
I20260928 10:57:50.813999 5709 ts_tablet_manager.cc:1428] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Time spent starting tablet: real 0.024s user 0.011s sys 0.009s
W20260928 10:57:50.841362 5552 tablet.cc:2369] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:57:50.852497 5413 tablet.cc:2369] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:50.852500 5717 raft_consensus.cc:482] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:50.852821 5717 raft_consensus.cc:504] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.854712 5717 leader_election.cc:288] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 323f35481c86457bb43f2dee5824fc7d (127.3.252.65:39911), e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67:46651)
I20260928 10:57:50.862905 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.855845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:53989 (local address 127.3.252.67:46651)
0928 10:57:50.856024 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.856058 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.856429 (+ 371us) server_negotiation.cc:407] Connection header received
0928 10:57:50.856580 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.856584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.856642 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.856723 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:50.857200 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.858101 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.860384 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.861593 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.861762 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.861866 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.862501 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.862634 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:57:50.862936 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.855077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:39601 (local address 127.3.252.65:39911)
0928 10:57:50.855242 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.855247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.855640 (+ 393us) server_negotiation.cc:407] Connection header received
0928 10:57:50.855907 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.855914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.855999 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.856105 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:50.856771 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.857709 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.860271 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.861593 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.861759 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.861866 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.862501 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.862634 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:57:50.863175 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.855730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:46651 (local address 127.3.252.66:53989)
0928 10:57:50.856325 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.856345 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.856467 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.856750 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.856754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.856779 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:50.857052 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.857059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.858217 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.858222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.860224 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.860234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.861931 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.861938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.862248 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.862365 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.862454 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.862682 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":166,"thread_start_us":273,"threads_started":1}
I20260928 10:57:50.863196 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.854972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39911 (local address 127.3.252.66:39601)
0928 10:57:50.855531 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:50.855552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:50.855700 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:50.856237 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:50.856242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:50.856278 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:50.856639 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.856648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.857938 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.857942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.860065 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:50.860082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.861931 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.861939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:50.862247 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.862362 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 10:57:50.862458 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 10:57:50.862682 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":133,"thread_start_us":264,"threads_started":1}
I20260928 10:57:50.864141 5367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b7a00b44e8b47e29f20c2c5940201df" candidate_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323f35481c86457bb43f2dee5824fc7d" is_pre_election: true
I20260928 10:57:50.864140 5644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b7a00b44e8b47e29f20c2c5940201df" candidate_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" is_pre_election: true
I20260928 10:57:50.864602 5367 raft_consensus.cc:2454] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1a2ed2a0449eb637758b3672e7c7 in term 0.
I20260928 10:57:50.864602 5644 raft_consensus.cc:2454] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1a2ed2a0449eb637758b3672e7c7 in term 0.
I20260928 10:57:50.865182 5433 leader_election.cc:302] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323f35481c86457bb43f2dee5824fc7d, c0fd1a2ed2a0449eb637758b3672e7c7; no voters:
I20260928 10:57:50.865545 5717 raft_consensus.cc:2781] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:50.865689 5717 raft_consensus.cc:482] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:50.865777 5717 raft_consensus.cc:3037] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:50.868083 5717 raft_consensus.cc:504] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.869124 5717 leader_election.cc:288] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [CANDIDATE]: Term 1 election: Requested vote from peers 323f35481c86457bb43f2dee5824fc7d (127.3.252.65:39911), e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67:46651)
I20260928 10:57:50.869586 5367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b7a00b44e8b47e29f20c2c5940201df" candidate_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323f35481c86457bb43f2dee5824fc7d"
I20260928 10:57:50.869637 5644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b7a00b44e8b47e29f20c2c5940201df" candidate_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0"
I20260928 10:57:50.869776 5367 raft_consensus.cc:3037] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:50.869918 5644 raft_consensus.cc:3037] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:50.872104 5367 raft_consensus.cc:2454] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1a2ed2a0449eb637758b3672e7c7 in term 1.
I20260928 10:57:50.872368 5644 raft_consensus.cc:2454] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1a2ed2a0449eb637758b3672e7c7 in term 1.
W20260928 10:57:50.872540 5691 tablet.cc:2369] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:50.872540 5433 leader_election.cc:302] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323f35481c86457bb43f2dee5824fc7d, c0fd1a2ed2a0449eb637758b3672e7c7; no voters:
I20260928 10:57:50.872815 5717 raft_consensus.cc:2781] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:50.873250 5717 raft_consensus.cc:686] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd1a2ed2a0449eb637758b3672e7c7, State: Running, Role: LEADER
I20260928 10:57:50.873670 5717 consensus_queue.cc:237] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } }
I20260928 10:57:50.878551 5204 catalog_manager.cc:5495] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } health_report { overall_health: HEALTHY } } }
I20260928 10:57:50.932164 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:50.926625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.3.252.66:33359)
0928 10:57:50.926845 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:50.926850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:50.926869 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:57:50.927146 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:50.927150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:50.927224 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:50.927339 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:50.927639 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.928388 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.930160 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:50.930550 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:50.930648 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:50.931073 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:50.931614 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:50.931720 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:50.931835 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 10:57:50.931901 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:57:50.940234 5644 raft_consensus.cc:1261] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1a2ed2a0449eb637758b3672e7c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:50.940265 5367 raft_consensus.cc:1261] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Refusing update from remote peer c0fd1a2ed2a0449eb637758b3672e7c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:50.941107 5720 consensus_queue.cc:1035] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:50.941457 5717 consensus_queue.cc:1035] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:50.953276 5723 mvcc.cc:205] Tried to move back new op lower bound from 7334269218561052672 to 7334269218302218240. Current Snapshot: MvccSnapshot[applied={T|T < 7334269218561052672}]
I20260928 10:57:50.971899 5205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:50.972164 5205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:50.975649 5205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871: Sending DeleteTablet for 3 replicas of tablet 0b7a00b44e8b47e29f20c2c5940201df
I20260928 10:57:50.976891 5342 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b7a00b44e8b47e29f20c2c5940201df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:50 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:39314
I20260928 10:57:50.976915 5481 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b7a00b44e8b47e29f20c2c5940201df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:50 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:47694
I20260928 10:57:50.977672 5620 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b7a00b44e8b47e29f20c2c5940201df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:50 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:43358
I20260928 10:57:50.978353 5204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:57:50.979069 5204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:50.980646 5735 tablet_replica.cc:330] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: stopping tablet replica
I20260928 10:57:50.981150 5735 raft_consensus.cc:2229] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:50.981660 5735 raft_consensus.cc:2258] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:50.987020 5620 tablet_service.cc:1467] Processing CreateTablet for tablet fa84c704650b4b58b743840e906a79d0 (DEFAULT_TABLE table=test-table [id=e9bc39caada647718b10b8d94d79902d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:50.987109 5341 tablet_service.cc:1467] Processing CreateTablet for tablet fa84c704650b4b58b743840e906a79d0 (DEFAULT_TABLE table=test-table [id=e9bc39caada647718b10b8d94d79902d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:50.987480 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa84c704650b4b58b743840e906a79d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:50.987594 5341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa84c704650b4b58b743840e906a79d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:50.988709 5480 tablet_service.cc:1467] Processing CreateTablet for tablet fa84c704650b4b58b743840e906a79d0 (DEFAULT_TABLE table=test-table [id=e9bc39caada647718b10b8d94d79902d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:50.989281 5480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa84c704650b4b58b743840e906a79d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:50.991170 5735 ts_tablet_manager.cc:1905] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:50.996881 5710 tablet_bootstrap.cc:492] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Bootstrap starting.
I20260928 10:57:50.998597 5710 tablet_bootstrap.cc:654] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:50.999233 5735 ts_tablet_manager.cc:1918] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:57:50.999413 5735 log.cc:1196] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/0b7a00b44e8b47e29f20c2c5940201df
I20260928 10:57:50.999969 5735 ts_tablet_manager.cc:1939] T 0b7a00b44e8b47e29f20c2c5940201df P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Deleting consensus metadata
I20260928 10:57:51.001245 5711 tablet_bootstrap.cc:492] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: Bootstrap starting.
I20260928 10:57:51.002424 5710 tablet_bootstrap.cc:492] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: No bootstrap required, opened a new log
I20260928 10:57:51.002593 5710 ts_tablet_manager.cc:1397] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 10:57:51.003078 5711 tablet_bootstrap.cc:654] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:51.003449 5710 raft_consensus.cc:348] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.003674 5710 raft_consensus.cc:374] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:51.003753 5710 raft_consensus.cc:729] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a73e7a7ac54f3a8f1fe4866c4b1ad0, State: Initialized, Role: FOLLOWER
I20260928 10:57:51.005293 5710 consensus_queue.cc:260] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.006016 5710 ts_tablet_manager.cc:1428] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 10:57:51.007545 5190 catalog_manager.cc:4841] TS e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67:46651): tablet 0b7a00b44e8b47e29f20c2c5940201df (table test-table [id=5fb015f3484b41db873b06f1305645a1]) successfully deleted
I20260928 10:57:51.009763 5711 tablet_bootstrap.cc:492] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: No bootstrap required, opened a new log
I20260928 10:57:51.009956 5711 ts_tablet_manager.cc:1397] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260928 10:57:51.010627 5709 tablet_bootstrap.cc:492] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: Bootstrap starting.
I20260928 10:57:51.010741 5711 raft_consensus.cc:348] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.010957 5711 raft_consensus.cc:374] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:51.011049 5711 raft_consensus.cc:729] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323f35481c86457bb43f2dee5824fc7d, State: Initialized, Role: FOLLOWER
I20260928 10:57:51.011343 5711 consensus_queue.cc:260] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.012326 5709 tablet_bootstrap.cc:654] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:51.015368 5711 ts_tablet_manager.cc:1428] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 10:57:51.025091 5709 tablet_bootstrap.cc:492] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: No bootstrap required, opened a new log
I20260928 10:57:51.025278 5709 ts_tablet_manager.cc:1397] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260928 10:57:51.026218 5709 raft_consensus.cc:348] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.026466 5709 raft_consensus.cc:374] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:51.026576 5709 raft_consensus.cc:729] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1a2ed2a0449eb637758b3672e7c7, State: Initialized, Role: FOLLOWER
I20260928 10:57:51.027115 5734 tablet_replica.cc:330] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: stopping tablet replica
I20260928 10:57:51.027623 5734 raft_consensus.cc:2229] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:51.026865 5709 consensus_queue.cc:260] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.028198 5734 raft_consensus.cc:2258] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:51.029060 5733 tablet_replica.cc:330] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: stopping tablet replica
I20260928 10:57:51.029487 5733 raft_consensus.cc:2229] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:57:51.030949 5709 ts_tablet_manager.cc:1428] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260928 10:57:51.031700 5734 ts_tablet_manager.cc:1905] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:51.034091 5733 raft_consensus.cc:2258] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:51.036243 5733 ts_tablet_manager.cc:1905] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:51.038470 5734 ts_tablet_manager.cc:1918] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:57:51.038637 5734 log.cc:1196] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/0b7a00b44e8b47e29f20c2c5940201df
I20260928 10:57:51.039135 5734 ts_tablet_manager.cc:1939] T 0b7a00b44e8b47e29f20c2c5940201df P 323f35481c86457bb43f2dee5824fc7d: Deleting consensus metadata
I20260928 10:57:51.040891 5190 catalog_manager.cc:4841] TS 323f35481c86457bb43f2dee5824fc7d (127.3.252.65:39911): tablet 0b7a00b44e8b47e29f20c2c5940201df (table test-table [id=5fb015f3484b41db873b06f1305645a1]) successfully deleted
I20260928 10:57:51.042611 5733 ts_tablet_manager.cc:1918] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:57:51.042735 5733 log.cc:1196] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/0b7a00b44e8b47e29f20c2c5940201df
I20260928 10:57:51.043188 5733 ts_tablet_manager.cc:1939] T 0b7a00b44e8b47e29f20c2c5940201df P c0fd1a2ed2a0449eb637758b3672e7c7: Deleting consensus metadata
I20260928 10:57:51.044688 5192 catalog_manager.cc:4841] TS c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66:33359): tablet 0b7a00b44e8b47e29f20c2c5940201df (table test-table [id=5fb015f3484b41db873b06f1305645a1]) successfully deleted
I20260928 10:57:51.047538 5715 raft_consensus.cc:482] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:51.047739 5715 raft_consensus.cc:504] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.049149 5715 leader_election.cc:288] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 323f35481c86457bb43f2dee5824fc7d (127.3.252.65:39911), c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66:33359)
I20260928 10:57:51.056241 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.050136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:53739 (local address 127.3.252.66:33359)
0928 10:57:51.050304 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.050309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.050326 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:57:51.050495 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.050499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.050586 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.050644 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:51.051335 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.052403 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.054490 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.055312 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.055460 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.055534 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.055824 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.055956 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:57:51.056411 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.049531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39911 (local address 127.3.252.67:49297)
0928 10:57:51.049810 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.049860 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.050004 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.050675 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.050680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.050709 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:51.050956 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.050968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.052017 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.052022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.053918 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.053933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.055650 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.055657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.055858 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.055893 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.055983 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.056120 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":3}
I20260928 10:57:51.056660 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.049619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:33359 (local address 127.3.252.67:53739)
0928 10:57:51.050205 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.050222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.050339 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.050788 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.050791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.050811 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:51.051170 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.051182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.052534 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.052539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.054345 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.054355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.055512 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.055518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.055675 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.055699 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.055777 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.055921 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":4,"thread_start_us":772,"threads_started":1}
I20260928 10:57:51.057063 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.049576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:49297 (local address 127.3.252.65:39911)
0928 10:57:51.049754 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.049760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.050127 (+ 367us) server_negotiation.cc:407] Connection header received
0928 10:57:51.050270 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.050284 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.050399 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.050504 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:51.051124 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.051892 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.054113 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.055296 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.055465 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.055556 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.056628 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.056780 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:57:51.057476 5501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa84c704650b4b58b743840e906a79d0" candidate_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" is_pre_election: true
I20260928 10:57:51.057683 5367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa84c704650b4b58b743840e906a79d0" candidate_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323f35481c86457bb43f2dee5824fc7d" is_pre_election: true
I20260928 10:57:51.057941 5367 raft_consensus.cc:2454] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a73e7a7ac54f3a8f1fe4866c4b1ad0 in term 0.
I20260928 10:57:51.057971 5501 raft_consensus.cc:2454] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a73e7a7ac54f3a8f1fe4866c4b1ad0 in term 0.
I20260928 10:57:51.058449 5572 leader_election.cc:302] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323f35481c86457bb43f2dee5824fc7d, e9a73e7a7ac54f3a8f1fe4866c4b1ad0; no voters:
I20260928 10:57:51.058861 5715 raft_consensus.cc:2781] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:51.059002 5715 raft_consensus.cc:482] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:51.059099 5715 raft_consensus.cc:3037] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:51.061498 5715 raft_consensus.cc:504] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.062186 5715 leader_election.cc:288] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [CANDIDATE]: Term 1 election: Requested vote from peers 323f35481c86457bb43f2dee5824fc7d (127.3.252.65:39911), c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66:33359)
I20260928 10:57:51.062717 5367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa84c704650b4b58b743840e906a79d0" candidate_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323f35481c86457bb43f2dee5824fc7d"
I20260928 10:57:51.062929 5367 raft_consensus.cc:3037] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:51.065296 5367 raft_consensus.cc:2454] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a73e7a7ac54f3a8f1fe4866c4b1ad0 in term 1.
I20260928 10:57:51.065641 5501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa84c704650b4b58b743840e906a79d0" candidate_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7"
I20260928 10:57:51.065721 5572 leader_election.cc:302] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323f35481c86457bb43f2dee5824fc7d, e9a73e7a7ac54f3a8f1fe4866c4b1ad0; no voters:
I20260928 10:57:51.065876 5501 raft_consensus.cc:3037] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:51.066071 5715 raft_consensus.cc:2781] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:51.066526 5715 raft_consensus.cc:686] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 LEADER]: Becoming Leader. State: Replica: e9a73e7a7ac54f3a8f1fe4866c4b1ad0, State: Running, Role: LEADER
I20260928 10:57:51.067178 5715 consensus_queue.cc:237] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } }
I20260928 10:57:51.068655 5501 raft_consensus.cc:2454] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a73e7a7ac54f3a8f1fe4866c4b1ad0 in term 1.
I20260928 10:57:51.071223 5205 catalog_manager.cc:5495] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } health_report { overall_health: HEALTHY } } }
Sep 28 10:57:51 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593068, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:51.126806 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.112383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36399 (local address 127.0.0.1:40606)
0928 10:57:51.112980 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.112998 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.113136 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.113441 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.113445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.114162 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:51.114424 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.114434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.115423 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.115427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.116404 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.116416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.117278 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.117284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.117434 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.118167 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:51.118188 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:51.120477 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:51.124286 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:51.124299 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:51.124309 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:51.124698 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:51.125054 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:51.125059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:51.125063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:51.125191 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:51.125704 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:51.125712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:51.125902 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.126052 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.126326 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":314,"threads_started":1}
I20260928 10:57:51.126833 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.112363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.3.252.126:36399)
0928 10:57:51.112545 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.112551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.113080 (+ 529us) server_negotiation.cc:407] Connection header received
0928 10:57:51.113241 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.113246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.113312 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.113390 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:51.114564 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.115293 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.116531 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.117061 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.117232 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.120830 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:51.120835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:51.120839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:51.120952 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:51.124155 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.124819 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.124824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.124828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.124955 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.125288 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.125293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.125296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.125583 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:51.125709 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.126147 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.126462 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":62}
I20260928 10:57:51.132062 5205 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40606:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:57:51.133057 5205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:51.139593 5620 tablet_service.cc:1467] Processing CreateTablet for tablet e41091d8c2d04986b23387056eb7910e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c87682c70b48489eae5f1239d2057956]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:57:51.139550 5481 tablet_service.cc:1467] Processing CreateTablet for tablet e41091d8c2d04986b23387056eb7910e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c87682c70b48489eae5f1239d2057956]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:57:51.139851 5342 tablet_service.cc:1467] Processing CreateTablet for tablet e41091d8c2d04986b23387056eb7910e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c87682c70b48489eae5f1239d2057956]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:57:51.140125 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41091d8c2d04986b23387056eb7910e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:51.140130 5481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41091d8c2d04986b23387056eb7910e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:51.140388 5342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41091d8c2d04986b23387056eb7910e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:51.147600 5711 tablet_bootstrap.cc:492] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: Bootstrap starting.
I20260928 10:57:51.148454 5710 tablet_bootstrap.cc:492] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Bootstrap starting.
I20260928 10:57:51.148828 5709 tablet_bootstrap.cc:492] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: Bootstrap starting.
I20260928 10:57:51.149295 5711 tablet_bootstrap.cc:654] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:51.150255 5710 tablet_bootstrap.cc:654] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:51.150866 5709 tablet_bootstrap.cc:654] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:51.159758 5711 tablet_bootstrap.cc:492] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: No bootstrap required, opened a new log
I20260928 10:57:51.160018 5711 ts_tablet_manager.cc:1397] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260928 10:57:51.160888 5711 raft_consensus.cc:348] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.161296 5711 raft_consensus.cc:374] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:51.161415 5711 raft_consensus.cc:729] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323f35481c86457bb43f2dee5824fc7d, State: Initialized, Role: FOLLOWER
I20260928 10:57:51.161756 5711 consensus_queue.cc:260] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.162882 5711 ts_tablet_manager.cc:1428] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 10:57:51.162729 5716 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.163694 5716 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:51.166124 5710 tablet_bootstrap.cc:492] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: No bootstrap required, opened a new log
I20260928 10:57:51.166306 5710 ts_tablet_manager.cc:1397] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20260928 10:57:51.166335 5709 tablet_bootstrap.cc:492] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: No bootstrap required, opened a new log
I20260928 10:57:51.166620 5709 ts_tablet_manager.cc:1397] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.002s
I20260928 10:57:51.167084 5710 raft_consensus.cc:348] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.167332 5710 raft_consensus.cc:374] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:51.167402 5710 raft_consensus.cc:729] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a73e7a7ac54f3a8f1fe4866c4b1ad0, State: Initialized, Role: FOLLOWER
I20260928 10:57:51.167686 5710 consensus_queue.cc:260] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.167747 5709 raft_consensus.cc:348] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.167936 5709 raft_consensus.cc:374] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:51.168026 5709 raft_consensus.cc:729] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1a2ed2a0449eb637758b3672e7c7, State: Initialized, Role: FOLLOWER
I20260928 10:57:51.168220 5710 ts_tablet_manager.cc:1428] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:57:51.168128 5715 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.168401 5715 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:51.168291 5709 consensus_queue.cc:260] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.168814 5728 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.169189 5728 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:51.170553 5709 ts_tablet_manager.cc:1428] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 10:57:51.418820 5716 raft_consensus.cc:482] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:51.419049 5716 raft_consensus.cc:504] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.420754 5716 leader_election.cc:288] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66:33359), e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67:46651)
I20260928 10:57:51.430869 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.421060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:33359 (local address 127.3.252.65:59625)
0928 10:57:51.422071 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.422093 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.422328 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.422871 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.422876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.422904 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:51.423305 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.423316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.424902 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.424907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.428063 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.428085 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.430015 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.430021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.430196 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.430231 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.430323 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.430466 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":7287,"threads_started":1}
I20260928 10:57:51.431253 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.421283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:59625 (local address 127.3.252.66:33359)
0928 10:57:51.421501 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.421508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.422192 (+ 684us) server_negotiation.cc:407] Connection header received
0928 10:57:51.422495 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.422500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.422575 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.422641 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:51.423513 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.424684 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.428280 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.429750 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.429985 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.430062 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.430372 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.430515 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:57:51.431905 5501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41091d8c2d04986b23387056eb7910e" candidate_uuid: "323f35481c86457bb43f2dee5824fc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" is_pre_election: true
I20260928 10:57:51.432206 5501 raft_consensus.cc:2454] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323f35481c86457bb43f2dee5824fc7d in term 0.
I20260928 10:57:51.432694 5295 leader_election.cc:302] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323f35481c86457bb43f2dee5824fc7d, c0fd1a2ed2a0449eb637758b3672e7c7; no voters:
I20260928 10:57:51.433145 5716 raft_consensus.cc:2781] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:51.433235 5716 raft_consensus.cc:482] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:51.433295 5716 raft_consensus.cc:3037] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:51.435199 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.421467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:41311 (local address 127.3.252.67:46651)
0928 10:57:51.427981 (+ 6514us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.427988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.428678 (+ 690us) server_negotiation.cc:407] Connection header received
0928 10:57:51.428834 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.428840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.428923 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.429005 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:51.429495 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.430524 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.432957 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.434179 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.434339 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.434411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.434767 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.434901 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6329,"thread_start_us":6148,"threads_started":1}
I20260928 10:57:51.435202 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.421120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:46651 (local address 127.3.252.65:41311)
0928 10:57:51.428578 (+ 7458us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.428594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.428717 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.429042 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.429047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.429073 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:51.429364 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.429372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.430671 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.430676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.432784 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.432797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.434409 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.434415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.434604 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.434638 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.434739 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.434884 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7348,"thread_start_us":585,"threads_started":1}
I20260928 10:57:51.436060 5716 raft_consensus.cc:504] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.436127 5644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41091d8c2d04986b23387056eb7910e" candidate_uuid: "323f35481c86457bb43f2dee5824fc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" is_pre_election: true
I20260928 10:57:51.436367 5644 raft_consensus.cc:2454] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323f35481c86457bb43f2dee5824fc7d in term 0.
I20260928 10:57:51.436692 5716 leader_election.cc:288] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66:33359), e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67:46651)
I20260928 10:57:51.437139 5501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41091d8c2d04986b23387056eb7910e" candidate_uuid: "323f35481c86457bb43f2dee5824fc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7"
I20260928 10:57:51.437387 5501 raft_consensus.cc:3037] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:51.437528 5644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41091d8c2d04986b23387056eb7910e" candidate_uuid: "323f35481c86457bb43f2dee5824fc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0"
I20260928 10:57:51.437717 5644 raft_consensus.cc:3037] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:51.439533 5501 raft_consensus.cc:2454] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323f35481c86457bb43f2dee5824fc7d in term 1.
I20260928 10:57:51.439904 5295 leader_election.cc:302] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323f35481c86457bb43f2dee5824fc7d, c0fd1a2ed2a0449eb637758b3672e7c7; no voters:
I20260928 10:57:51.440194 5716 raft_consensus.cc:2781] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:51.440632 5716 raft_consensus.cc:686] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [term 1 LEADER]: Becoming Leader. State: Replica: 323f35481c86457bb43f2dee5824fc7d, State: Running, Role: LEADER
I20260928 10:57:51.441043 5644 raft_consensus.cc:2454] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323f35481c86457bb43f2dee5824fc7d in term 1.
I20260928 10:57:51.441056 5716 consensus_queue.cc:237] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } }
I20260928 10:57:51.443217 5754 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: TxnStatusTablet state changed. Reason: New leader 323f35481c86457bb43f2dee5824fc7d. Latest consensus state: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.443593 5754 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:57:51.444911 5204 catalog_manager.cc:5495] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 323f35481c86457bb43f2dee5824fc7d (127.3.252.65). New cstate: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } health_report { overall_health: HEALTHY } } }
I20260928 10:57:51.446530 5756 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:57:51.446735 5756 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:57:51.528829 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.523149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39911 (local address 127.0.0.1:39330)
0928 10:57:51.523387 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.523404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.523519 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.524002 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.524007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.524019 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:51.524192 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.524201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.525209 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.525215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.526268 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.526280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.527105 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.527111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.527310 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.527375 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:51.528296 (+ 921us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:51.528316 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.528397 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.528553 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 10:57:51.528842 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.523438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.3.252.65:39911)
0928 10:57:51.523624 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.523631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.523650 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:57:51.523759 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.523763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.523843 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.523962 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:51.524321 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.525087 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.526399 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.526961 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.527083 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.527561 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:51.528191 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:51.528316 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.528418 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.528498 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:57:51.533427 5644 raft_consensus.cc:1261] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Refusing update from remote peer 323f35481c86457bb43f2dee5824fc7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:51.533638 5501 raft_consensus.cc:1261] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Refusing update from remote peer 323f35481c86457bb43f2dee5824fc7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:51.534240 5716 consensus_queue.cc:1035] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:51.534633 5754 consensus_queue.cc:1035] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:51.542773 5715 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: TxnStatusTablet state changed. Reason: New leader 323f35481c86457bb43f2dee5824fc7d. Latest consensus state: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.543056 5715 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:51.546245 5759 mvcc.cc:205] Tried to move back new op lower bound from 7334269220994031616 to 7334269220626706432. Current Snapshot: MvccSnapshot[applied={T|T < 7334269220994031616}]
I20260928 10:57:51.546568 5728 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: TxnStatusTablet state changed. Reason: New leader 323f35481c86457bb43f2dee5824fc7d. Latest consensus state: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.546819 5728 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:51.550344 5754 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.550604 5754 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:57:51.550895 5754 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.551122 5754 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P 323f35481c86457bb43f2dee5824fc7d: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:57:51.555603 5715 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.555868 5715 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:51.557978 5728 tablet_replica.cc:439] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "323f35481c86457bb43f2dee5824fc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 } } peers { permanent_uuid: "e9a73e7a7ac54f3a8f1fe4866c4b1ad0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 46651 } } peers { permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 } } }
I20260928 10:57:51.558218 5728 tablet_replica.cc:442] T e41091d8c2d04986b23387056eb7910e P c0fd1a2ed2a0449eb637758b3672e7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:51.568832 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.562777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.3.252.67:46651)
0928 10:57:51.562975 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.562981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.563450 (+ 469us) server_negotiation.cc:407] Connection header received
0928 10:57:51.563644 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.563648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.563735 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.563865 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:51.564314 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.564968 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.566743 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.567205 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.567320 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.567721 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:51.568259 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:51.568380 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.568527 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.568601 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 10:57:51.582561 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.577429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:40889 (local address 127.3.252.65:39911)
0928 10:57:51.577574 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.577580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.578021 (+ 441us) server_negotiation.cc:407] Connection header received
0928 10:57:51.578196 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.578200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.578263 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.578340 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:51.578737 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.579438 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.580752 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.581311 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.581414 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.581768 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:51.581935 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:51.582011 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.582140 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.582212 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 10:57:51.582566 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.577397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39911 (local address 127.3.252.67:40889)
0928 10:57:51.577932 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.577948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.578068 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.578380 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.578385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.578396 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:51.578597 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.578605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.579563 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.579569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.580632 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.580643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.581443 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.581449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.581574 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.581622 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:51.582021 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:51.582038 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.582105 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.582230 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":261,"threads_started":1}
I20260928 10:57:51.591886 5501 raft_consensus.cc:1261] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Refusing update from remote peer e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:51.591997 5367 raft_consensus.cc:1261] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Refusing update from remote peer e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:51.592559 5715 consensus_queue.cc:1035] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1a2ed2a0449eb637758b3672e7c7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:51.593580 5772 consensus_queue.cc:1035] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "323f35481c86457bb43f2dee5824fc7d" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:51.602556 5760 mvcc.cc:205] Tried to move back new op lower bound from 7334269221234536448 to 7334269219096047616. Current Snapshot: MvccSnapshot[applied={T|T < 7334269221234536448}]
I20260928 10:57:51.648458 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.631558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:33217 (local address 127.3.252.67:46651)
0928 10:57:51.631978 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.631984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.642600 (+ 10616us) server_negotiation.cc:407] Connection header received
0928 10:57:51.642918 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.642925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.643032 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.643101 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:51.643872 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.644746 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.646623 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.647070 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.647176 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.647674 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:51.647805 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:51.647887 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.648125 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.648203 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20260928 10:57:51.649267 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.631648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:46651 (local address 127.3.252.65:33217)
0928 10:57:51.642506 (+ 10858us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:51.642527 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:51.642718 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:51.643283 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:51.643287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:51.643299 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:51.643686 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.643701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.644881 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.644884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.646455 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:51.646467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.647233 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.647240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:51.647400 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.647478 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:51.647991 (+ 513us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:51.648001 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 10:57:51.648686 (+ 685us) client_negotiation.cc:240] Negotiation successful
0928 10:57:51.648818 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10613,"thread_start_us":10389,"threads_started":1}
I20260928 10:57:51.707510 5204 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:51.707782 5204 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:51.711309 5204 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6364ec1e19a4350ac2c4544dfb65871: Sending DeleteTablet for 3 replicas of tablet fa84c704650b4b58b743840e906a79d0
I20260928 10:57:51.712312 5620 tablet_service.cc:1514] Processing DeleteTablet for tablet fa84c704650b4b58b743840e906a79d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:51 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:43358
I20260928 10:57:51.712738 5481 tablet_service.cc:1514] Processing DeleteTablet for tablet fa84c704650b4b58b743840e906a79d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:51 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:47694
I20260928 10:57:51.713227 5342 tablet_service.cc:1514] Processing DeleteTablet for tablet fa84c704650b4b58b743840e906a79d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:51 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:39314
I20260928 10:57:51.734316 5784 tablet_replica.cc:330] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: stopping tablet replica
I20260928 10:57:51.734582 5784 raft_consensus.cc:2229] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:51.734879 5784 raft_consensus.cc:2258] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:51.735117 5782 tablet_replica.cc:330] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: stopping tablet replica
I20260928 10:57:51.735368 5782 raft_consensus.cc:2229] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:57:51.736006 5782 raft_consensus.cc:2258] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:51.737571 5784 ts_tablet_manager.cc:1905] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:51.742854 5783 tablet_replica.cc:330] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: stopping tablet replica
I20260928 10:57:51.743155 5783 raft_consensus.cc:2229] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:51.743433 5782 ts_tablet_manager.cc:1905] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:51.743482 5783 raft_consensus.cc:2258] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:51.744679 5783 ts_tablet_manager.cc:1905] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:51.751338 5782 ts_tablet_manager.cc:1918] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:51.751495 5782 log.cc:1196] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/fa84c704650b4b58b743840e906a79d0
I20260928 10:57:51.751914 5783 ts_tablet_manager.cc:1918] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:51.752125 5782 ts_tablet_manager.cc:1939] T fa84c704650b4b58b743840e906a79d0 P e9a73e7a7ac54f3a8f1fe4866c4b1ad0: Deleting consensus metadata
I20260928 10:57:51.752159 5783 log.cc:1196] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/fa84c704650b4b58b743840e906a79d0
I20260928 10:57:51.752717 5783 ts_tablet_manager.cc:1939] T fa84c704650b4b58b743840e906a79d0 P c0fd1a2ed2a0449eb637758b3672e7c7: Deleting consensus metadata
I20260928 10:57:51.753465 5190 catalog_manager.cc:4841] TS e9a73e7a7ac54f3a8f1fe4866c4b1ad0 (127.3.252.67:46651): tablet fa84c704650b4b58b743840e906a79d0 (table test-table [id=e9bc39caada647718b10b8d94d79902d]) successfully deleted
I20260928 10:57:51.754052 5192 catalog_manager.cc:4841] TS c0fd1a2ed2a0449eb637758b3672e7c7 (127.3.252.66:33359): tablet fa84c704650b4b58b743840e906a79d0 (table test-table [id=e9bc39caada647718b10b8d94d79902d]) successfully deleted
I20260928 10:57:51.757795 5784 ts_tablet_manager.cc:1918] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:51.757992 5784 log.cc:1196] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/fa84c704650b4b58b743840e906a79d0
I20260928 10:57:51.758450 5784 ts_tablet_manager.cc:1939] T fa84c704650b4b58b743840e906a79d0 P 323f35481c86457bb43f2dee5824fc7d: Deleting consensus metadata
I20260928 10:57:51.759563 5190 catalog_manager.cc:4841] TS 323f35481c86457bb43f2dee5824fc7d (127.3.252.65:39911): tablet fa84c704650b4b58b743840e906a79d0 (table test-table [id=e9bc39caada647718b10b8d94d79902d]) successfully deleted
Sep 28 10:57:51 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:57:51.770545 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.737101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.3.252.65:39911)
0928 10:57:51.737274 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.737279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.742663 (+ 5384us) server_negotiation.cc:407] Connection header received
0928 10:57:51.742987 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.742993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.743076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.743148 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:51.745506 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.748104 (+ 2598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.755827 (+ 7723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.756531 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.756643 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.762525 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:51.762540 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:51.762565 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:51.762621 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:51.766727 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.767885 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.767896 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.767899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.768074 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.768525 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.768530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.768534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.769022 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:51.769355 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.769665 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.769948 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 10:57:51.771284 5407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39338
I20260928 10:57:51.799275 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.775107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.3.252.126:36399)
0928 10:57:51.775583 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.775590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.782540 (+ 6950us) server_negotiation.cc:407] Connection header received
0928 10:57:51.782621 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.782627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.782715 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.782846 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:51.784940 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.786219 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.788029 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.788677 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.788800 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.792214 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:51.792222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:51.792227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:51.792273 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:51.795861 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.796742 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.796748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.796752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.796910 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.797373 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.797378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.797381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.797751 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:51.798030 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.798456 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.798700 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":199,"threads_started":1}
W20260928 10:57:51.800319 5204 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40610
I20260928 10:57:51.802711 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5276
I20260928 10:57:51.822007 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.804626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.3.252.65:39911)
0928 10:57:51.804784 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.804790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.804812 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:57:51.805032 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.805038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.805118 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.805184 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:51.806859 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.807999 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.809760 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.810582 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.810694 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.813667 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:51.813675 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:51.813679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:51.813732 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:51.817529 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.818369 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.818375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.818379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.818561 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.818957 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.818962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.818966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.819383 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:51.819612 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.821333 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.821612 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 10:57:51.823354 5407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39354
W20260928 10:57:51.823907 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5276: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:57:51.824086 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 5276
I20260928 10:57:51.849038 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5416
Sep 28 10:57:51 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:57:51.869222 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.849794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.3.252.66:33359)
0928 10:57:51.850030 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.850035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.850055 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:57:51.850181 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.850187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.850252 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.850329 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:51.851452 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.852181 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.853401 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.853919 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.854036 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.863247 (+ 9211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:51.863263 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:51.863280 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:51.863319 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:51.866183 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.866906 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.866915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.866917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.867044 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.867404 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.867409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.867411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.867740 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:51.868068 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.868763 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.868928 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260928 10:57:51.869962 5546 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47710
W20260928 10:57:51.870658 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5416: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:57:51.870821 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 5416
I20260928 10:57:51.888718 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5555
Sep 28 10:57:51 dist-test-slave-djjw krb5kdc[5160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:57:51.904459 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:51.889441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.3.252.67:46651)
0928 10:57:51.889620 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:51.889625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:51.889710 (+ 85us) server_negotiation.cc:407] Connection header received
0928 10:57:51.889972 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:51.889976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:51.890066 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:51.890156 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:51.891329 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.892102 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.893321 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:51.893778 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:51.893938 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:51.898042 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:51.898063 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:51.898090 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:51.898139 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:51.901704 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.902386 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.902396 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.902400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.902559 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:51.902906 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:51.902911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:51.902915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:51.903420 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:51.903714 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:51.903881 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 10:57:51.904096 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 10:57:51.905102 5685 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43384
W20260928 10:57:51.905721 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5555: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:57:51.905889 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 5555
I20260928 10:57:51.920645 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5173
I20260928 10:57:51.921752 5245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:52.032719 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 5173
2026-09-28T10:57:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4623 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:52 dist-test-slave-djjw krb5kdc[5794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:52 dist-test-slave-djjw krb5kdc[5794](info): set up 2 sockets
Sep 28 10:57:52 dist-test-slave-djjw krb5kdc[5794](info): commencing operation
krb5kdc: starting...
W20260928 10:57:53.029997 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:57:53 dist-test-slave-djjw krb5kdc[5794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:57:53.208871 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33439
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:53.523797 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:53.524099 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:53.524250 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:53.531402 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:53.531512 5807 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:57:53.531551 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:53.531610 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:53.531657 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:53.531715 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:53.541990 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:33439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:53.543246 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:53.544507 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:53.556068 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:53.556155 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:53.556155 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:53.556667 5807 server_base.cc:1034] running on GCE node
I20260928 10:57:53.557432 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:53.558876 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:53.560099 5807 hybrid_clock.cc:648] HybridClock initialized: now 1790593073560022 us; error 51 us; skew 500 ppm
I20260928 10:57:53.560402 5807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:53 dist-test-slave-djjw krb5kdc[5794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:53.563571 5807 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:57:53.565114 5807 webserver.cc:460] Webserver started at http://127.3.252.126:43317/ using document root <none> and password file <none>
I20260928 10:57:53.565994 5807 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:53.566104 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:53.566443 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:53.569566 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "d0abb362af7e4b4c9c7d5733b6d9e659"
format_stamp: "Formatted at 2026-09-28 10:57:53 on dist-test-slave-djjw"
server_key: "549bbcce36d9bc948369d3a1ff94d0df"
server_key_iv: "0c045571e6654a48e968b51b7ed98bb1"
server_key_version: "encryptionkey@0"
I20260928 10:57:53.570309 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "d0abb362af7e4b4c9c7d5733b6d9e659"
format_stamp: "Formatted at 2026-09-28 10:57:53 on dist-test-slave-djjw"
server_key: "549bbcce36d9bc948369d3a1ff94d0df"
server_key_iv: "0c045571e6654a48e968b51b7ed98bb1"
server_key_version: "encryptionkey@0"
I20260928 10:57:53.575387 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:57:53.578883 5822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:53.580224 5807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:57:53.580394 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "d0abb362af7e4b4c9c7d5733b6d9e659"
format_stamp: "Formatted at 2026-09-28 10:57:53 on dist-test-slave-djjw"
server_key: "549bbcce36d9bc948369d3a1ff94d0df"
server_key_iv: "0c045571e6654a48e968b51b7ed98bb1"
server_key_version: "encryptionkey@0"
I20260928 10:57:53.580585 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:53.592760 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:53.596839 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:53.597088 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:53.616055 5807 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:33439
I20260928 10:57:53.616061 5883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:33439 every 8 connection(s)
I20260928 10:57:53.617374 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:53.621124 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:53.625463 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 5807
I20260928 10:57:53.625638 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:57:53.626385 4081 external_mini_cluster.cc:1428] Setting key 7eb196e41cf396bea943f98bd5befaf5
I20260928 10:57:53.632167 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Bootstrap starting.
I20260928 10:57:53.635842 5884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:53.637090 5884 log.cc:824] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:53.640429 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: No bootstrap required, opened a new log
Sep 28 10:57:53 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:53.647992 5884 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33439 } }
I20260928 10:57:53.648381 5884 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:53.648486 5884 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0abb362af7e4b4c9c7d5733b6d9e659, State: Initialized, Role: FOLLOWER
I20260928 10:57:53.649170 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:53.628452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.3.252.126:33439)
0928 10:57:53.629228 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:53.629243 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:53.629288 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:57:53.631029 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:53.631054 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:53.631484 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:53.631806 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:53.633575 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:53.634956 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:53.636686 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:53.637453 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:53.637633 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:53.642072 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:53.642083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:53.642100 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:53.642147 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:53.645089 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:53.645763 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:53.645775 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:53.645785 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:53.645990 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:53.646346 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:53.646351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:53.646355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:53.646926 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:53.647214 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:53.647608 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 10:57:53.648034 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":325,"threads_started":1}
I20260928 10:57:53.649093 5884 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33439 } }
I20260928 10:57:53.649344 5884 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:53.649485 5884 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:53.649653 5884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:53.652243 5884 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33439 } }
I20260928 10:57:53.652705 5884 leader_election.cc:302] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0abb362af7e4b4c9c7d5733b6d9e659; no voters:
I20260928 10:57:53.653235 5884 leader_election.cc:288] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:53.653375 5889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:53.654186 5889 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [term 1 LEADER]: Becoming Leader. State: Replica: d0abb362af7e4b4c9c7d5733b6d9e659, State: Running, Role: LEADER
I20260928 10:57:53.654721 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33439 } }
I20260928 10:57:53.655330 5884 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:53.657119 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0abb362af7e4b4c9c7d5733b6d9e659. Latest consensus state: current_term: 1 leader_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33439 } } }
I20260928 10:57:53.657420 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:53.658248 5896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:53.658115 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0abb362af7e4b4c9c7d5733b6d9e659" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33439 } } }
I20260928 10:57:53.658399 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:53.664007 5896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:53.672364 5896 catalog_manager.cc:1347] Generated new cluster ID: 985bfd563082439dbb27514fe864821b
I20260928 10:57:53.672472 5896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:53.682961 5896 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:53.684391 5896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:53.695995 5896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Generated new TSK 0
I20260928 10:57:53.696877 5896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:57:53.743288 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33439
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:54.054860 5910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:54.055150 5910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:54.055276 5910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:54.062620 5910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:54.062731 5910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:54.062860 5910 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:57:54.062943 5910 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:57:54.063041 5910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:57:54.072753 5910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:33439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:54.074076 5910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:54.075395 5910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:54.088028 5915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:54.087967 5918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:54.088009 5916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:54.088973 5910 server_base.cc:1034] running on GCE node
I20260928 10:57:54.089560 5910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:54.090525 5910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:54.091733 5910 hybrid_clock.cc:648] HybridClock initialized: now 1790593074091681 us; error 45 us; skew 500 ppm
I20260928 10:57:54.092038 5910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:54 dist-test-slave-djjw krb5kdc[5794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:54.095012 5910 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:57:54.096395 5910 webserver.cc:460] Webserver started at http://127.3.252.65:45025/ using document root <none> and password file <none>
I20260928 10:57:54.097023 5910 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:54.097122 5910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:54.097369 5910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:54.100246 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "a41796cd662d4c6da3e572ec21d2b5ca"
format_stamp: "Formatted at 2026-09-28 10:57:54 on dist-test-slave-djjw"
server_key: "1540ae1ed3963127efc8e92780a378ef"
server_key_iv: "4e570cfb34c22ee93a4c665aefea8ebe"
server_key_version: "encryptionkey@0"
I20260928 10:57:54.101037 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "a41796cd662d4c6da3e572ec21d2b5ca"
format_stamp: "Formatted at 2026-09-28 10:57:54 on dist-test-slave-djjw"
server_key: "1540ae1ed3963127efc8e92780a378ef"
server_key_iv: "4e570cfb34c22ee93a4c665aefea8ebe"
server_key_version: "encryptionkey@0"
I20260928 10:57:54.106338 5910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:57:54.109990 5925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:54.111454 5910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:57:54.111610 5910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "a41796cd662d4c6da3e572ec21d2b5ca"
format_stamp: "Formatted at 2026-09-28 10:57:54 on dist-test-slave-djjw"
server_key: "1540ae1ed3963127efc8e92780a378ef"
server_key_iv: "4e570cfb34c22ee93a4c665aefea8ebe"
server_key_version: "encryptionkey@0"
I20260928 10:57:54.111799 5910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:54.127918 5910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:54.132217 5910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:54.132445 5910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:54.135035 5910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:54.135138 5910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:54.135250 5910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:54.135344 5910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:57:54 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:54.157269 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.135552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:43341 (local address 127.3.252.126:33439)
0928 10:57:54.135750 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:54.135756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:54.137120 (+ 1364us) server_negotiation.cc:407] Connection header received
0928 10:57:54.139171 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:54.139176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:54.139267 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:54.139361 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:54.141630 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.142448 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.145191 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.145725 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.145877 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.150451 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:54.150456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:54.150460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:54.150492 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:54.153628 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.154630 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.154637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.154640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.154805 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.155264 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.155269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.155273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.155645 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:54.155810 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:54.156850 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0928 10:57:54.157022 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 10:57:54.158941 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.136096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33439 (local address 127.3.252.65:43341)
0928 10:57:54.136938 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:54.136986 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:54.138886 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:54.139763 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:54.139772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:54.140593 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:54.141446 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.141471 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.142620 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.142625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.145008 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.145023 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.146002 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.146008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.146306 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.147121 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:54.147151 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:54.150214 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:54.153767 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.153776 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.153798 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.154482 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.154915 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.154918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.154922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.155134 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.155891 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:54.155901 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:54.156425 (+ 524us) client_negotiation.cc:769] Sending connection context
0928 10:57:54.156750 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 10:57:54.157310 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":343,"threads_started":1}
I20260928 10:57:54.181766 5910 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:43295
I20260928 10:57:54.181792 6044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:43295 every 8 connection(s)
I20260928 10:57:54.183416 5910 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:57:54.185583 6045 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33439
I20260928 10:57:54.185781 6045 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:54.186450 6045 heartbeater.cc:511] Master 127.3.252.126:33439 requested a full tablet report, sending...
I20260928 10:57:54.186939 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.172677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:43223 (local address 127.3.252.126:33439)
0928 10:57:54.172885 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:54.172892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:54.173453 (+ 561us) server_negotiation.cc:407] Connection header received
0928 10:57:54.173644 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:54.173648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:54.173714 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:54.173800 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:54.175004 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.175703 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.177760 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.178295 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.178420 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.180681 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:54.180688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:54.180694 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:54.180735 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:54.183882 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.184883 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.184888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.184892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.185088 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.185471 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.185476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.185478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.185813 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:54.185986 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:54.186385 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 10:57:54.186602 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:57:54.187039 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.172654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33439 (local address 127.3.252.65:43223)
0928 10:57:54.173307 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:54.173331 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:54.173508 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:54.173879 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:54.173884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:54.174473 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:54.174864 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.174877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.175855 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.175861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.177591 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.177612 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.178663 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.178670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.178853 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.179499 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:54.179524 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:54.180486 (+ 962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:54.184026 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.184035 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.184039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.184690 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.185207 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.185212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.185215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.185360 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.185952 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:54.185957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:54.186159 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 10:57:54.186282 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 10:57:54.186612 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":322,"threads_started":1}
I20260928 10:57:54.188292 5839 ts_manager.cc:194] Registered new tserver with Master: a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295)
I20260928 10:57:54.189659 5839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:43341
I20260928 10:57:54.192076 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 5910
I20260928 10:57:54.192263 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:57:54.192852 4081 external_mini_cluster.cc:1428] Setting key 3f6a8434f9bc1b0dc5e2c30daa8952c5
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:57:54.228545 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33439
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:54.538625 6049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:54.538908 6049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:54.539008 6049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:54.546159 6049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:54.546265 6049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:54.546353 6049 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:57:54.546402 6049 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:57:54.546468 6049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:57:54.556134 6049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:33439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:54.557411 6049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:54.558686 6049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:54.572463 6055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:54.572446 6054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:54.573027 6057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:54.573649 6049 server_base.cc:1034] running on GCE node
I20260928 10:57:54.574299 6049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:54.575240 6049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:54.576442 6049 hybrid_clock.cc:648] HybridClock initialized: now 1790593074576396 us; error 42 us; skew 500 ppm
I20260928 10:57:54.576756 6049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:54 dist-test-slave-djjw krb5kdc[5794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:54.579885 6049 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:57:54.581465 6049 webserver.cc:460] Webserver started at http://127.3.252.66:46593/ using document root <none> and password file <none>
I20260928 10:57:54.582309 6049 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:54.582428 6049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:54.582722 6049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:54.585817 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "a569538ff0af4513b0e31c050c76e091"
format_stamp: "Formatted at 2026-09-28 10:57:54 on dist-test-slave-djjw"
server_key: "9f7a796502f5aed3b8646c152f2f5f16"
server_key_iv: "4174f56d58b20acae7086e9febbb8a62"
server_key_version: "encryptionkey@0"
I20260928 10:57:54.586674 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "a569538ff0af4513b0e31c050c76e091"
format_stamp: "Formatted at 2026-09-28 10:57:54 on dist-test-slave-djjw"
server_key: "9f7a796502f5aed3b8646c152f2f5f16"
server_key_iv: "4174f56d58b20acae7086e9febbb8a62"
server_key_version: "encryptionkey@0"
I20260928 10:57:54.592154 6049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:57:54.595861 6064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:54.597258 6049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:57:54.597429 6049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "a569538ff0af4513b0e31c050c76e091"
format_stamp: "Formatted at 2026-09-28 10:57:54 on dist-test-slave-djjw"
server_key: "9f7a796502f5aed3b8646c152f2f5f16"
server_key_iv: "4174f56d58b20acae7086e9febbb8a62"
server_key_version: "encryptionkey@0"
I20260928 10:57:54.597620 6049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:54.610891 6049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:54.614831 6049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:54.615087 6049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:54.617802 6049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:54.617931 6049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:54.618099 6049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:54.618185 6049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:57:54 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:54.642611 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.618667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:46665 (local address 127.3.252.126:33439)
0928 10:57:54.618850 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:54.618856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:54.620706 (+ 1850us) server_negotiation.cc:407] Connection header received
0928 10:57:54.623597 (+ 2891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:54.623604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:54.623701 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:54.623811 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:54.626655 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.627891 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.630499 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.631194 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.631294 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.635891 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:54.635897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:54.635902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:54.635943 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:54.638720 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.639738 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.639745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.639749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.639897 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.640426 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.640432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.640436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.640804 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:54.640961 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:54.641997 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0928 10:57:54.642257 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 10:57:54.644189 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.619631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33439 (local address 127.3.252.66:46665)
0928 10:57:54.620524 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:54.620578 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:54.623232 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:54.624242 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:54.624259 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:54.625356 (+ 1097us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:54.626396 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.626432 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.628117 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.628122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.630312 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.630328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.631648 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.631655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.631935 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.632739 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:54.632769 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:54.635631 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:54.638860 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.638870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.638890 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.639570 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.640007 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.640012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.640024 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.640290 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.640941 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:54.640948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:54.641479 (+ 531us) client_negotiation.cc:769] Sending connection context
0928 10:57:54.641869 (+ 390us) client_negotiation.cc:240] Negotiation successful
0928 10:57:54.642464 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":364,"threads_started":1}
I20260928 10:57:54.664263 6049 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:40255
I20260928 10:57:54.664278 6183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:40255 every 8 connection(s)
I20260928 10:57:54.665670 6049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:57:54.666726 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.651580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:46725 (local address 127.3.252.126:33439)
0928 10:57:54.651767 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:54.651772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:54.652377 (+ 605us) server_negotiation.cc:407] Connection header received
0928 10:57:54.652570 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:54.652573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:54.652637 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:54.652726 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:54.653818 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.654926 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.657134 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.657748 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.657901 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.660450 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:54.660457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:54.660463 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:54.660504 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:54.663771 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.664642 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.664646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.664649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.664763 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:54.665207 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:54.665212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:54.665216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:54.665546 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:54.665668 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:54.666152 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 10:57:54.666385 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:57:54.666853 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:54.651571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33439 (local address 127.3.252.66:46725)
0928 10:57:54.652256 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:54.652279 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:54.652450 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:54.652781 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:54.652786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:54.653400 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:54.653675 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.653684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.655121 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.655127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.656991 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:54.657006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:54.658000 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:54.658008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:54.658226 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:54.658946 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:54.658975 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:54.660241 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:54.663910 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.663915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.663919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.664520 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.664938 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:54.664943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:54.664947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:54.665105 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:54.665683 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:54.665688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:54.665918 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 10:57:54.666047 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 10:57:54.666409 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":363,"threads_started":1}
I20260928 10:57:54.667889 6184 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33439
I20260928 10:57:54.668175 6184 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:54.668979 6184 heartbeater.cc:511] Master 127.3.252.126:33439 requested a full tablet report, sending...
I20260928 10:57:54.670357 5839 ts_manager.cc:194] Registered new tserver with Master: a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255)
I20260928 10:57:54.671275 5839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:46665
I20260928 10:57:54.675477 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 6049
I20260928 10:57:54.675621 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:57:54.676005 4081 external_mini_cluster.cc:1428] Setting key b550534f28df84f9924e463f0505753c
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:57:54.713670 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33439
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:55.025899 6188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:55.026204 6188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:55.026345 6188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:55.033679 6188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:55.033812 6188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:55.033947 6188 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:57:55.034015 6188 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:57:55.034119 6188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:57:55.044293 6188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:33439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:55.045600 6188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:55.046983 6188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:55.060954 6193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:55.061028 6194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:55.060989 6196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:55.063154 6188 server_base.cc:1034] running on GCE node
I20260928 10:57:55.063730 6188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:55.064670 6188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:55.065918 6188 hybrid_clock.cc:648] HybridClock initialized: now 1790593075065860 us; error 41 us; skew 500 ppm
I20260928 10:57:55.066251 6188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:55 dist-test-slave-djjw krb5kdc[5794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593075, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:55.069301 6188 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:57:55.070751 6188 webserver.cc:460] Webserver started at http://127.3.252.67:33227/ using document root <none> and password file <none>
I20260928 10:57:55.071417 6188 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:55.071525 6188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:55.071802 6188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:55.074640 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "ed945e883e7344c5aac653508dbd96a7"
format_stamp: "Formatted at 2026-09-28 10:57:55 on dist-test-slave-djjw"
server_key: "9ce66722f4411928ccfeb41179f843c8"
server_key_iv: "4d35e7e5eec74794a4e1219aee44e416"
server_key_version: "encryptionkey@0"
I20260928 10:57:55.075353 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "ed945e883e7344c5aac653508dbd96a7"
format_stamp: "Formatted at 2026-09-28 10:57:55 on dist-test-slave-djjw"
server_key: "9ce66722f4411928ccfeb41179f843c8"
server_key_iv: "4d35e7e5eec74794a4e1219aee44e416"
server_key_version: "encryptionkey@0"
I20260928 10:57:55.080308 6188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:57:55.083565 6203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:55.084894 6188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:57:55.085029 6188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "ed945e883e7344c5aac653508dbd96a7"
format_stamp: "Formatted at 2026-09-28 10:57:55 on dist-test-slave-djjw"
server_key: "9ce66722f4411928ccfeb41179f843c8"
server_key_iv: "4d35e7e5eec74794a4e1219aee44e416"
server_key_version: "encryptionkey@0"
I20260928 10:57:55.085193 6188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:55.107034 6188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:55.110795 6188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:55.111057 6188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:55.113577 6188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:57:55.113677 6188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:55.113785 6188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:57:55.113888 6188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:57:55 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593075, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:55.136544 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.114280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:57031 (local address 127.3.252.126:33439)
0928 10:57:55.114477 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.114483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.115893 (+ 1410us) server_negotiation.cc:407] Connection header received
0928 10:57:55.118210 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.118217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.118298 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.118372 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:55.120793 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.121706 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.124013 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.124561 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.124680 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.129496 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:55.129500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:55.129504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:55.129536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:55.132892 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.133857 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.133863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.133866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.134074 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.134623 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.134628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.134632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.135013 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:55.135169 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.135886 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.136156 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:57:55.137300 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.114857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33439 (local address 127.3.252.67:57031)
0928 10:57:55.115724 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.115770 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.117869 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.118873 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.118885 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.119738 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:55.120608 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.120632 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.121900 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.121905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.123819 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.123838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.125016 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.125022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.125300 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.126188 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:55.126230 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:55.129271 (+ 3041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:55.133062 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:55.133075 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:55.133098 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:55.133660 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:55.134225 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:55.134230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:55.134234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:55.134485 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:55.135153 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:55.135161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:55.135524 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.135761 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.136151 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":369,"threads_started":1}
I20260928 10:57:55.157183 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.143370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33439 (local address 127.3.252.67:47337)
0928 10:57:55.143896 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.143910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.144042 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.144407 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.144413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.145011 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:55.145393 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.145404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.146747 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.146753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.147965 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.147978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.149083 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.149089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.149303 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.150019 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:55.150046 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:55.151154 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:55.154409 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:55.154415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:55.154419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:55.155059 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:55.155478 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:55.155483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:55.155487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:55.155612 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:55.156118 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:55.156123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:55.156301 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.156458 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.156818 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260928 10:57:55.157316 6188 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:40889
I20260928 10:57:55.157512 6322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:40889 every 8 connection(s)
I20260928 10:57:55.158004 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.143298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:47337 (local address 127.3.252.126:33439)
0928 10:57:55.143460 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.143464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.143989 (+ 525us) server_negotiation.cc:407] Connection header received
0928 10:57:55.144170 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.144175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.144255 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.144349 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:55.145545 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.146578 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.148180 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.148734 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.148848 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.151318 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:55.151324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:55.151329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:55.151373 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:55.154250 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.155209 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.155215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.155219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.155370 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.155709 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.155714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.155717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.156010 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:55.156132 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.157507 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.157719 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:57:55.158855 6188 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:57:55.160990 6323 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33439
I20260928 10:57:55.161214 6323 heartbeater.cc:464] Registering TS with master...
I20260928 10:57:55.161777 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 6188
I20260928 10:57:55.161967 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:57:55.162101 6323 heartbeater.cc:511] Master 127.3.252.126:33439 requested a full tablet report, sending...
I20260928 10:57:55.162505 4081 external_mini_cluster.cc:1428] Setting key b6cc4d08de6b3302e6d49e3b53d269e2
I20260928 10:57:55.163480 5839 ts_manager.cc:194] Registered new tserver with Master: ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889)
I20260928 10:57:55.164444 5839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:57031
I20260928 10:57:55.165292 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:57:55 dist-test-slave-djjw krb5kdc[5794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 10:57:55.192299 6045 heartbeater.cc:503] Master 127.3.252.126:33439 was elected leader, sending a full tablet report...
Sep 28 10:57:55 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:55.208779 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.196193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.3.252.126:33439)
0928 10:57:55.196340 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.196344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.196601 (+ 257us) server_negotiation.cc:407] Connection header received
0928 10:57:55.196738 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.196741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.196819 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.196940 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:55.198246 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.198957 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.200170 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.200566 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.200659 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.204266 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:55.204271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:55.204274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:55.204308 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:55.206663 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.207264 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.207268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.207270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.207383 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.207730 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.207733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.207735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.207969 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:55.208137 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.208403 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.208564 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:57:55.212767 5838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:57:55.219705 5838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:55.253420 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.231634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.3.252.65:43295)
0928 10:57:55.242589 (+ 10955us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.242598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.243816 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 10:57:55.243935 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.243944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.244251 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.244439 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:55.245450 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.247114 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.250563 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.252042 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.252201 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.252432 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.252924 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.253111 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10779,"thread_start_us":10971,"threads_started":1}
I20260928 10:57:55.253520 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.231424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43295 (local address 127.0.0.1:40118)
0928 10:57:55.243331 (+ 11907us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.243388 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.243590 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.244727 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.244736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.244769 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:55.245241 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.245254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.247301 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.247307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.250368 (+ 3061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.250381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.252319 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.252326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.252546 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.252609 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.252756 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.253065 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11652,"mutex_wait_us":61,"thread_start_us":17310,"threads_started":1}
I20260928 10:57:55.257570 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.233066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40889 (local address 127.0.0.1:55976)
0928 10:57:55.249031 (+ 15965us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.249053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.249278 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.250137 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.250142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.250168 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:55.250569 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.250578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.251989 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.251994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.254697 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.254711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.256658 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.256665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.256871 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.256907 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.256997 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.257155 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15824,"mutex_wait_us":849,"thread_start_us":15110,"threads_started":1}
I20260928 10:57:55.257756 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.233652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.3.252.67:40889)
0928 10:57:55.240391 (+ 6739us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.240403 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.249171 (+ 8768us) server_negotiation.cc:407] Connection header received
0928 10:57:55.249470 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.249479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.249733 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.249942 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:55.250756 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.251815 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.254852 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.256135 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.256264 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.256498 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.257264 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.257436 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6529,"thread_start_us":6311,"threads_started":1}
I20260928 10:57:55.258363 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.231749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.3.252.66:40255)
0928 10:57:55.247381 (+ 15632us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.247391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.248599 (+ 1208us) server_negotiation.cc:407] Connection header received
0928 10:57:55.248775 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.248784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.249080 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.249292 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:55.250203 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.251613 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.255619 (+ 4006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.256994 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.257112 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.257296 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.257810 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.258105 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15214,"thread_start_us":15402,"threads_started":1}
I20260928 10:57:55.258206 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1704eacfc4b35b7df6a45e31410eb (DEFAULT_TABLE table=test-table [id=74713b9a7b6d4f5a89db800de36e509d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:55.259057 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.231602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:40255 (local address 127.0.0.1:40608)
0928 10:57:55.248453 (+ 16851us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.248479 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.248622 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.249517 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.249522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.249544 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:55.250011 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.250025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.251764 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.251769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.255433 (+ 3664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.255450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.257412 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.257419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.257603 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.257646 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.258602 (+ 956us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.258753 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16722,"thread_start_us":11348,"threads_started":1}
I20260928 10:57:55.259670 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1704eacfc4b35b7df6a45e31410eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.262084 6114 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1704eacfc4b35b7df6a45e31410eb (DEFAULT_TABLE table=test-table [id=74713b9a7b6d4f5a89db800de36e509d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:55.261989 6253 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1704eacfc4b35b7df6a45e31410eb (DEFAULT_TABLE table=test-table [id=74713b9a7b6d4f5a89db800de36e509d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:55.263311 6114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1704eacfc4b35b7df6a45e31410eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.263415 6253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1704eacfc4b35b7df6a45e31410eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.273553 6342 tablet_bootstrap.cc:492] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Bootstrap starting.
I20260928 10:57:55.275646 6343 tablet_bootstrap.cc:492] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Bootstrap starting.
I20260928 10:57:55.278057 6342 tablet_bootstrap.cc:654] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.278266 6344 tablet_bootstrap.cc:492] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Bootstrap starting.
I20260928 10:57:55.279486 6343 tablet_bootstrap.cc:654] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.279443 6342 log.cc:824] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:55.280566 6343 log.cc:824] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:55.281224 6344 tablet_bootstrap.cc:654] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.282372 6344 log.cc:824] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:55.282795 6342 tablet_bootstrap.cc:492] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: No bootstrap required, opened a new log
I20260928 10:57:55.283178 6342 ts_tablet_manager.cc:1397] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260928 10:57:55.284792 6344 tablet_bootstrap.cc:492] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: No bootstrap required, opened a new log
I20260928 10:57:55.285120 6344 ts_tablet_manager.cc:1397] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 10:57:55.288582 6343 tablet_bootstrap.cc:492] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: No bootstrap required, opened a new log
I20260928 10:57:55.288923 6343 ts_tablet_manager.cc:1397] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260928 10:57:55.292748 6344 raft_consensus.cc:348] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.293128 6344 raft_consensus.cc:374] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.293251 6344 raft_consensus.cc:729] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed945e883e7344c5aac653508dbd96a7, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.293424 6342 raft_consensus.cc:348] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.293936 6342 raft_consensus.cc:374] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.293845 6344 consensus_queue.cc:260] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.294035 6342 raft_consensus.cc:729] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a569538ff0af4513b0e31c050c76e091, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.294889 6323 heartbeater.cc:503] Master 127.3.252.126:33439 was elected leader, sending a full tablet report...
I20260928 10:57:55.295404 6344 ts_tablet_manager.cc:1428] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260928 10:57:55.294625 6342 consensus_queue.cc:260] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.297132 6184 heartbeater.cc:503] Master 127.3.252.126:33439 was elected leader, sending a full tablet report...
I20260928 10:57:55.297753 6342 ts_tablet_manager.cc:1428] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260928 10:57:55.300024 6343 raft_consensus.cc:348] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.300459 6343 raft_consensus.cc:374] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.300561 6343 raft_consensus.cc:729] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41796cd662d4c6da3e572ec21d2b5ca, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.301165 6343 consensus_queue.cc:260] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.302696 6343 ts_tablet_manager.cc:1428] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
W20260928 10:57:55.410512 6324 tablet.cc:2369] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:55.415796 6348 raft_consensus.cc:482] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:55.416181 6348 raft_consensus.cc:504] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.417898 6348 leader_election.cc:288] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255)
W20260928 10:57:55.418582 6185 tablet.cc:2369] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:55.424549 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.418226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:39259 (local address 127.3.252.65:43295)
0928 10:57:55.418444 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.418450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.418727 (+ 277us) server_negotiation.cc:407] Connection header received
0928 10:57:55.418817 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.418822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.418893 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.418976 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:55.419576 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.420432 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.422684 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.423739 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.423891 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.423963 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.424205 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.424340 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:57:55.424548 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.418131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43295 (local address 127.3.252.67:39259)
0928 10:57:55.418432 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.418452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.418596 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.419145 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.419153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.419181 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:55.419382 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.419395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.420603 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.420607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.422530 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.422545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.423884 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.423888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.424062 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.424086 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.424158 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.424251 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":54}
I20260928 10:57:55.425159 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.418759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:36495 (local address 127.3.252.66:40255)
0928 10:57:55.419339 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.419345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.419363 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:57:55.419590 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.419595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.419668 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.419754 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:55.420244 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.420994 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.423130 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.424205 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.424354 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.424416 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.424797 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.424919 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449}
I20260928 10:57:55.425206 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.418260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:40255 (local address 127.3.252.67:36495)
0928 10:57:55.419274 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.419291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.419412 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.419781 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.419784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.419804 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:55.420098 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.420106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.421139 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.421144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.422985 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.422995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.424420 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.424427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.424600 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.424629 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.424754 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.424868 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":2,"thread_start_us":707,"threads_started":1}
I20260928 10:57:55.425606 5996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1704eacfc4b35b7df6a45e31410eb" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" is_pre_election: true
I20260928 10:57:55.426105 5996 raft_consensus.cc:2454] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 0.
I20260928 10:57:55.426069 6134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1704eacfc4b35b7df6a45e31410eb" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a569538ff0af4513b0e31c050c76e091" is_pre_election: true
I20260928 10:57:55.426486 6134 raft_consensus.cc:2454] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 0.
I20260928 10:57:55.426753 6207 leader_election.cc:302] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41796cd662d4c6da3e572ec21d2b5ca, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:55.427112 6348 raft_consensus.cc:2781] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:55.427230 6348 raft_consensus.cc:482] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:55.427321 6348 raft_consensus.cc:3037] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.429389 6348 raft_consensus.cc:504] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.429983 6348 leader_election.cc:288] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 election: Requested vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255)
I20260928 10:57:55.430338 5996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1704eacfc4b35b7df6a45e31410eb" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca"
I20260928 10:57:55.430408 6134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1704eacfc4b35b7df6a45e31410eb" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a569538ff0af4513b0e31c050c76e091"
I20260928 10:57:55.430533 5996 raft_consensus.cc:3037] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.430583 6134 raft_consensus.cc:3037] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.432663 6134 raft_consensus.cc:2454] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 1.
I20260928 10:57:55.432674 5996 raft_consensus.cc:2454] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 1.
I20260928 10:57:55.433027 6206 leader_election.cc:302] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a569538ff0af4513b0e31c050c76e091, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:55.433321 6348 raft_consensus.cc:2781] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:55.433728 6348 raft_consensus.cc:686] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 1 LEADER]: Becoming Leader. State: Replica: ed945e883e7344c5aac653508dbd96a7, State: Running, Role: LEADER
I20260928 10:57:55.434222 6348 consensus_queue.cc:237] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
W20260928 10:57:55.436139 6046 tablet.cc:2369] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:55.438745 5838 catalog_manager.cc:5495] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed945e883e7344c5aac653508dbd96a7 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "ed945e883e7344c5aac653508dbd96a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } health_report { overall_health: HEALTHY } } }
Sep 28 10:57:55 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:55.493129 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.480273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33439 (local address 127.0.0.1:45802)
0928 10:57:55.480805 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.480819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.480958 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.481270 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.481273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.481967 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:55.482215 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.482224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.483280 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.483285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.484288 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.484299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.485101 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.485108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.485289 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.485874 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:55.485892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:55.487947 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:55.490709 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:55.490718 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:55.490724 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:55.491097 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:55.491460 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:55.491463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:55.491465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:55.491580 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:55.492101 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:55.492107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:55.492246 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.492376 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.492592 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":303,"threads_started":1}
I20260928 10:57:55.493463 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.480245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.3.252.126:33439)
0928 10:57:55.480416 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.480422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.480902 (+ 480us) server_negotiation.cc:407] Connection header received
0928 10:57:55.481072 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.481076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.481137 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.481222 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:55.482359 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.483165 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.484418 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.484927 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.485015 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.488110 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:55.488115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:55.488118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:55.488146 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:55.490599 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.491238 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.491242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.491244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.491367 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:55.491684 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:55.491687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:55.491689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:55.492000 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:55.492107 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.492484 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.492746 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":59}
I20260928 10:57:55.499806 5838 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:45802:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:57:55.500954 5838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:55.508162 6114 tablet_service.cc:1467] Processing CreateTablet for tablet b4832ed0dabb4f09af96bbc2f80b3897 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2203e856a0a94b9f92f3ec808eca4ee2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:57:55.508248 6253 tablet_service.cc:1467] Processing CreateTablet for tablet b4832ed0dabb4f09af96bbc2f80b3897 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2203e856a0a94b9f92f3ec808eca4ee2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:57:55.508158 5975 tablet_service.cc:1467] Processing CreateTablet for tablet b4832ed0dabb4f09af96bbc2f80b3897 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2203e856a0a94b9f92f3ec808eca4ee2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:57:55.509007 6114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4832ed0dabb4f09af96bbc2f80b3897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.509017 6253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4832ed0dabb4f09af96bbc2f80b3897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.509017 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4832ed0dabb4f09af96bbc2f80b3897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.515676 6342 tablet_bootstrap.cc:492] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: Bootstrap starting.
I20260928 10:57:55.515767 6344 tablet_bootstrap.cc:492] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: Bootstrap starting.
I20260928 10:57:55.517076 6344 tablet_bootstrap.cc:654] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.517078 6342 tablet_bootstrap.cc:654] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.519374 6342 tablet_bootstrap.cc:492] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: No bootstrap required, opened a new log
I20260928 10:57:55.519515 6342 ts_tablet_manager.cc:1397] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 10:57:55.520356 6343 tablet_bootstrap.cc:492] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: Bootstrap starting.
I20260928 10:57:55.520270 6342 raft_consensus.cc:348] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.520517 6342 raft_consensus.cc:374] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.520598 6342 raft_consensus.cc:729] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a569538ff0af4513b0e31c050c76e091, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.520881 6342 consensus_queue.cc:260] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.521358 6344 tablet_bootstrap.cc:492] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: No bootstrap required, opened a new log
I20260928 10:57:55.521510 6344 ts_tablet_manager.cc:1397] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 10:57:55.521317 6349 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.521622 6349 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:55.522042 6343 tablet_bootstrap.cc:654] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.522248 6344 raft_consensus.cc:348] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.522493 6344 raft_consensus.cc:374] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.522578 6344 raft_consensus.cc:729] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed945e883e7344c5aac653508dbd96a7, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.522742 6342 ts_tablet_manager.cc:1428] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 10:57:55.522778 6344 consensus_queue.cc:260] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.523306 6344 ts_tablet_manager.cc:1428] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 10:57:55.523901 6348 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.524204 6348 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:55.531509 6343 tablet_bootstrap.cc:492] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: No bootstrap required, opened a new log
I20260928 10:57:55.531692 6343 ts_tablet_manager.cc:1397] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260928 10:57:55.532464 6343 raft_consensus.cc:348] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.532716 6343 raft_consensus.cc:374] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.532806 6343 raft_consensus.cc:729] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41796cd662d4c6da3e572ec21d2b5ca, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.533026 6343 consensus_queue.cc:260] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.533596 6343 ts_tablet_manager.cc:1428] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 10:57:55.533485 6350 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.533921 6350 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:55.610546 6350 raft_consensus.cc:482] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:55.610812 6350 raft_consensus.cc:504] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.612315 6350 leader_election.cc:288] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255), ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889)
I20260928 10:57:55.618986 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.612788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:60957 (local address 127.3.252.67:40889)
0928 10:57:55.613176 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.613182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.613723 (+ 541us) server_negotiation.cc:407] Connection header received
0928 10:57:55.613905 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.613911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.613966 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.614049 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:55.614539 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.615287 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.617063 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.618093 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.618251 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.618323 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.618614 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.618735 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260928 10:57:55.619088 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.612764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40889 (local address 127.3.252.65:60957)
0928 10:57:55.613626 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.613640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.613762 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.614079 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.614083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.614109 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:55.614387 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.614394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.615409 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.615413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.616921 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.616932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.618276 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.618282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.618453 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.618488 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.618576 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.618712 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":1,"thread_start_us":565,"threads_started":1}
I20260928 10:57:55.619927 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.612815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:42549 (local address 127.3.252.66:40255)
0928 10:57:55.613007 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.613013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.613409 (+ 396us) server_negotiation.cc:407] Connection header received
0928 10:57:55.613602 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.613606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.613664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.613734 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:55.614299 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.615018 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.617447 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.618563 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.618707 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.618766 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.619557 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.619676 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:57:55.619927 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.612611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:40255 (local address 127.3.252.65:42549)
0928 10:57:55.613306 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.613321 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.613468 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.613772 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.613776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.613801 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:55.614144 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.614153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.615146 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.615151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.617276 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.617291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.618913 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.618918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.619401 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.619424 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.619515 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.619627 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":396,"threads_started":1}
I20260928 10:57:55.619910 6273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4832ed0dabb4f09af96bbc2f80b3897" candidate_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed945e883e7344c5aac653508dbd96a7" is_pre_election: true
I20260928 10:57:55.620261 6273 raft_consensus.cc:2454] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41796cd662d4c6da3e572ec21d2b5ca in term 0.
I20260928 10:57:55.620558 6134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4832ed0dabb4f09af96bbc2f80b3897" candidate_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a569538ff0af4513b0e31c050c76e091" is_pre_election: true
I20260928 10:57:55.620786 6134 raft_consensus.cc:2454] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41796cd662d4c6da3e572ec21d2b5ca in term 0.
I20260928 10:57:55.620752 5926 leader_election.cc:302] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41796cd662d4c6da3e572ec21d2b5ca, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:55.621157 6350 raft_consensus.cc:2781] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:55.621281 6350 raft_consensus.cc:482] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:55.621373 6350 raft_consensus.cc:3037] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.624413 6350 raft_consensus.cc:504] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.625051 6350 leader_election.cc:288] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [CANDIDATE]: Term 1 election: Requested vote from peers a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255), ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889)
I20260928 10:57:55.625535 6273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4832ed0dabb4f09af96bbc2f80b3897" candidate_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed945e883e7344c5aac653508dbd96a7"
I20260928 10:57:55.625524 6134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4832ed0dabb4f09af96bbc2f80b3897" candidate_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a569538ff0af4513b0e31c050c76e091"
I20260928 10:57:55.625716 6273 raft_consensus.cc:3037] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.625726 6134 raft_consensus.cc:3037] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.627962 6273 raft_consensus.cc:2454] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41796cd662d4c6da3e572ec21d2b5ca in term 1.
I20260928 10:57:55.628396 5926 leader_election.cc:302] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41796cd662d4c6da3e572ec21d2b5ca, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:55.628530 6134 raft_consensus.cc:2454] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41796cd662d4c6da3e572ec21d2b5ca in term 1.
I20260928 10:57:55.628657 6350 raft_consensus.cc:2781] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:55.629169 6350 raft_consensus.cc:686] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 LEADER]: Becoming Leader. State: Replica: a41796cd662d4c6da3e572ec21d2b5ca, State: Running, Role: LEADER
I20260928 10:57:55.629705 6350 consensus_queue.cc:237] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.633041 6365 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: TxnStatusTablet state changed. Reason: New leader a41796cd662d4c6da3e572ec21d2b5ca. Latest consensus state: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.633476 6365 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:57:55.634155 6367 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:57:55.634428 6367 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:57:55.634819 5838 catalog_manager.cc:5495] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca reported cstate change: term changed from 0 to 1, leader changed from <none> to a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65). New cstate: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260928 10:57:55.681250 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.675162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43295 (local address 127.0.0.1:40122)
0928 10:57:55.675414 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.675434 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.675566 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.676068 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.676073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.676086 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:55.676273 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.676283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.677725 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.677729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.678732 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.678743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.679651 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.679655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.679801 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.679852 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:55.680815 (+ 963us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:55.680831 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.680882 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.681026 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 10:57:55.681310 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.675367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.3.252.65:43295)
0928 10:57:55.675556 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.675562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.675586 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:57:55.675778 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.675783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.675885 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.676027 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:55.676414 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.677580 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.678864 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.679488 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.679624 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.680035 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:55.680709 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:55.680855 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.680917 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.680995 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:57:55.689729 6273 raft_consensus.cc:1261] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Refusing update from remote peer a41796cd662d4c6da3e572ec21d2b5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:55.689901 6134 raft_consensus.cc:1261] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Refusing update from remote peer a41796cd662d4c6da3e572ec21d2b5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:55.690713 6350 consensus_queue.cc:1035] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:55.690989 6365 consensus_queue.cc:1035] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:55.699591 6348 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: TxnStatusTablet state changed. Reason: New leader a41796cd662d4c6da3e572ec21d2b5ca. Latest consensus state: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.699877 6348 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:55.702227 6370 mvcc.cc:205] Tried to move back new op lower bound from 7334269238015995904 to 7334269237783216128. Current Snapshot: MvccSnapshot[applied={T|T < 7334269238015995904}]
I20260928 10:57:55.704095 6349 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: TxnStatusTablet state changed. Reason: New leader a41796cd662d4c6da3e572ec21d2b5ca. Latest consensus state: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.704365 6349 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:55.705058 6371 mvcc.cc:205] Tried to move back new op lower bound from 7334269238015995904 to 7334269237783216128. Current Snapshot: MvccSnapshot[applied={T|T < 7334269238015995904 or (T in {7334269238015995904})}]
I20260928 10:57:55.706054 6348 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.706321 6348 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P ed945e883e7344c5aac653508dbd96a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:55.707288 6365 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.707551 6365 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:57:55.712872 6372 mvcc.cc:205] Tried to move back new op lower bound from 7334269238015995904 to 7334269237783216128. Current Snapshot: MvccSnapshot[applied={T|T < 7334269238015995904}]
I20260928 10:57:55.713565 6350 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.713943 6350 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P a41796cd662d4c6da3e572ec21d2b5ca: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:57:55.724848 6349 tablet_replica.cc:439] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } } }
I20260928 10:57:55.725108 6349 tablet_replica.cc:442] T b4832ed0dabb4f09af96bbc2f80b3897 P a569538ff0af4513b0e31c050c76e091: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:57:55.735934 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.725730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.3.252.67:40889)
0928 10:57:55.725989 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.725995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.730693 (+ 4698us) server_negotiation.cc:407] Connection header received
0928 10:57:55.730999 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.731004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.731104 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.731233 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:55.731685 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.732429 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.733822 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.734248 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.734359 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.734786 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:55.735333 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:55.735460 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.735571 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.735656 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 10:57:55.751969 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.746086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43295 (local address 127.3.252.67:50269)
0928 10:57:55.746626 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.746642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.746769 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.747117 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.747120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.747129 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:55.747419 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.747427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.748815 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.748819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.749892 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.749904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.750883 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.750897 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.751016 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.751074 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:55.751480 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:55.751487 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.751543 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.751646 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":250,"threads_started":1}
I20260928 10:57:55.751965 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.746199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:50269 (local address 127.3.252.65:43295)
0928 10:57:55.746389 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.746395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.746734 (+ 339us) server_negotiation.cc:407] Connection header received
0928 10:57:55.746894 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.746900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.747017 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.747108 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:55.747569 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.748708 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.750022 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.750694 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.750823 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.751256 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:55.751386 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:55.751478 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.751577 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.751661 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:57:55.762008 6134 raft_consensus.cc:1261] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Refusing update from remote peer ed945e883e7344c5aac653508dbd96a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:55.762367 5996 raft_consensus.cc:1261] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Refusing update from remote peer ed945e883e7344c5aac653508dbd96a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:55.762600 6352 consensus_queue.cc:1035] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:55.763217 6348 consensus_queue.cc:1035] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:55.814572 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.808667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:35725 (local address 127.3.252.67:40889)
0928 10:57:55.808823 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:55.808829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:55.808986 (+ 157us) server_negotiation.cc:407] Connection header received
0928 10:57:55.809231 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:55.809236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:55.809305 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:55.809375 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:55.810006 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.810972 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.812409 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.812923 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.813040 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.813772 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:55.813934 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:55.814021 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:55.814285 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 10:57:55.814361 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:57:55.814819 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:55.808231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40889 (local address 127.3.252.65:35725)
0928 10:57:55.808883 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:55.808913 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:55.809104 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:55.809416 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:55.809420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:55.809431 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:55.809787 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.809799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.811102 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.811107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.812276 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:55.812287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:55.813118 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:55.813125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:55.813487 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:55.813570 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:55.814163 (+ 593us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:55.814175 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:57:55.814257 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 10:57:55.814398 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":661,"threads_started":1}
I20260928 10:57:55.861044 5839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:55.861323 5839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:55.865263 5839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Sending DeleteTablet for 3 replicas of tablet 1ac1704eacfc4b35b7df6a45e31410eb
I20260928 10:57:55.866499 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1704eacfc4b35b7df6a45e31410eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:55 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40118
I20260928 10:57:55.866511 6114 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1704eacfc4b35b7df6a45e31410eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:55 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40608
I20260928 10:57:55.867767 6253 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1704eacfc4b35b7df6a45e31410eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:55 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55976
I20260928 10:57:55.875834 5839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:57:55.876577 5839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:55.889403 6252 tablet_service.cc:1467] Processing CreateTablet for tablet 5be8ef9cd7b74a418a11beca66aca0e5 (DEFAULT_TABLE table=test-table [id=7e1d094b436e45dd8468aa336d393018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:55.889489 6113 tablet_service.cc:1467] Processing CreateTablet for tablet 5be8ef9cd7b74a418a11beca66aca0e5 (DEFAULT_TABLE table=test-table [id=7e1d094b436e45dd8468aa336d393018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:55.890005 6252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be8ef9cd7b74a418a11beca66aca0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.890040 6113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be8ef9cd7b74a418a11beca66aca0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.893260 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 5be8ef9cd7b74a418a11beca66aca0e5 (DEFAULT_TABLE table=test-table [id=7e1d094b436e45dd8468aa336d393018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:55.893795 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be8ef9cd7b74a418a11beca66aca0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:55.896425 6344 tablet_bootstrap.cc:492] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: Bootstrap starting.
I20260928 10:57:55.896725 6342 tablet_bootstrap.cc:492] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: Bootstrap starting.
I20260928 10:57:55.898150 6344 tablet_bootstrap.cc:654] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.898329 6342 tablet_bootstrap.cc:654] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.904240 6395 tablet_replica.cc:330] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: stopping tablet replica
I20260928 10:57:55.904685 6395 raft_consensus.cc:2229] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:57:55.904786 6343 tablet_bootstrap.cc:492] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: Bootstrap starting.
I20260928 10:57:55.906420 6395 raft_consensus.cc:2258] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:55.906539 6343 tablet_bootstrap.cc:654] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:55.907126 6393 tablet_replica.cc:330] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: stopping tablet replica
I20260928 10:57:55.907507 6393 raft_consensus.cc:2229] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:55.907869 6393 raft_consensus.cc:2258] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:55.908612 6394 tablet_replica.cc:330] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: stopping tablet replica
I20260928 10:57:55.908998 6394 raft_consensus.cc:2229] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:55.909293 6394 raft_consensus.cc:2258] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:55.910904 6344 tablet_bootstrap.cc:492] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: No bootstrap required, opened a new log
I20260928 10:57:55.911067 6344 ts_tablet_manager.cc:1397] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20260928 10:57:55.911852 6344 raft_consensus.cc:348] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.912094 6344 raft_consensus.cc:374] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.912158 6344 raft_consensus.cc:729] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed945e883e7344c5aac653508dbd96a7, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.912400 6344 consensus_queue.cc:260] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.912921 6344 ts_tablet_manager.cc:1428] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 10:57:55.914738 6395 ts_tablet_manager.cc:1905] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:55.917954 6343 tablet_bootstrap.cc:492] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: No bootstrap required, opened a new log
I20260928 10:57:55.918124 6343 ts_tablet_manager.cc:1397] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260928 10:57:55.919063 6342 tablet_bootstrap.cc:492] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: No bootstrap required, opened a new log
I20260928 10:57:55.918944 6343 raft_consensus.cc:348] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.919188 6343 raft_consensus.cc:374] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.919269 6343 raft_consensus.cc:729] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41796cd662d4c6da3e572ec21d2b5ca, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.919441 6342 ts_tablet_manager.cc:1397] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260928 10:57:55.919515 6394 ts_tablet_manager.cc:1905] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:55.919564 6343 consensus_queue.cc:260] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.920769 6342 raft_consensus.cc:348] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.921059 6342 raft_consensus.cc:374] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:55.921164 6342 raft_consensus.cc:729] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a569538ff0af4513b0e31c050c76e091, State: Initialized, Role: FOLLOWER
I20260928 10:57:55.921478 6342 consensus_queue.cc:260] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.923928 6390 raft_consensus.cc:482] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:55.924180 6343 ts_tablet_manager.cc:1428] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260928 10:57:55.924081 6390 raft_consensus.cc:504] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.924796 6390 leader_election.cc:288] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255)
I20260928 10:57:55.925374 6342 ts_tablet_manager.cc:1428] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260928 10:57:55.925552 6133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8ef9cd7b74a418a11beca66aca0e5" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a569538ff0af4513b0e31c050c76e091" is_pre_election: true
I20260928 10:57:55.925849 6133 raft_consensus.cc:2454] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 0.
I20260928 10:57:55.926404 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8ef9cd7b74a418a11beca66aca0e5" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" is_pre_election: true
I20260928 10:57:55.926666 5994 raft_consensus.cc:2454] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 0.
I20260928 10:57:55.927161 6207 leader_election.cc:302] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41796cd662d4c6da3e572ec21d2b5ca, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:55.929286 6390 raft_consensus.cc:2781] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:55.929402 6390 raft_consensus.cc:482] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:55.929490 6390 raft_consensus.cc:3037] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.931077 6395 ts_tablet_manager.cc:1918] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:55.931237 6395 log.cc:1196] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/1ac1704eacfc4b35b7df6a45e31410eb
I20260928 10:57:55.931645 6395 ts_tablet_manager.cc:1939] T 1ac1704eacfc4b35b7df6a45e31410eb P ed945e883e7344c5aac653508dbd96a7: Deleting consensus metadata
I20260928 10:57:55.932300 6390 raft_consensus.cc:504] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.933094 5823 catalog_manager.cc:4841] TS ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889): tablet 1ac1704eacfc4b35b7df6a45e31410eb (table test-table [id=74713b9a7b6d4f5a89db800de36e509d]) successfully deleted
I20260928 10:57:55.933358 6133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8ef9cd7b74a418a11beca66aca0e5" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a569538ff0af4513b0e31c050c76e091"
I20260928 10:57:55.933548 6133 raft_consensus.cc:3037] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.934068 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8ef9cd7b74a418a11beca66aca0e5" candidate_uuid: "ed945e883e7344c5aac653508dbd96a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca"
I20260928 10:57:55.934267 5994 raft_consensus.cc:3037] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:55.936424 6133 raft_consensus.cc:2454] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 1.
I20260928 10:57:55.937225 6206 leader_election.cc:302] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a569538ff0af4513b0e31c050c76e091, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:55.937367 5994 raft_consensus.cc:2454] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed945e883e7344c5aac653508dbd96a7 in term 1.
I20260928 10:57:55.937548 6348 raft_consensus.cc:2781] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:55.937705 6348 raft_consensus.cc:686] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 1 LEADER]: Becoming Leader. State: Replica: ed945e883e7344c5aac653508dbd96a7, State: Running, Role: LEADER
I20260928 10:57:55.938027 6348 consensus_queue.cc:237] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:55.939142 6390 leader_election.cc:288] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [CANDIDATE]: Term 1 election: Requested vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255)
I20260928 10:57:55.939481 6393 ts_tablet_manager.cc:1905] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:55.950636 5838 catalog_manager.cc:5495] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed945e883e7344c5aac653508dbd96a7 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "ed945e883e7344c5aac653508dbd96a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260928 10:57:55.951308 6393 ts_tablet_manager.cc:1918] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:55.951479 6393 log.cc:1196] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/1ac1704eacfc4b35b7df6a45e31410eb
I20260928 10:57:55.951995 6393 ts_tablet_manager.cc:1939] T 1ac1704eacfc4b35b7df6a45e31410eb P a569538ff0af4513b0e31c050c76e091: Deleting consensus metadata
I20260928 10:57:55.953696 6394 ts_tablet_manager.cc:1918] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:55.953810 5825 catalog_manager.cc:4841] TS a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255): tablet 1ac1704eacfc4b35b7df6a45e31410eb (table test-table [id=74713b9a7b6d4f5a89db800de36e509d]) successfully deleted
I20260928 10:57:55.953927 6394 log.cc:1196] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/1ac1704eacfc4b35b7df6a45e31410eb
I20260928 10:57:55.954687 6394 ts_tablet_manager.cc:1939] T 1ac1704eacfc4b35b7df6a45e31410eb P a41796cd662d4c6da3e572ec21d2b5ca: Deleting consensus metadata
I20260928 10:57:55.956650 5826 catalog_manager.cc:4841] TS a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295): tablet 1ac1704eacfc4b35b7df6a45e31410eb (table test-table [id=74713b9a7b6d4f5a89db800de36e509d]) successfully deleted
I20260928 10:57:55.993501 5994 raft_consensus.cc:1261] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Refusing update from remote peer ed945e883e7344c5aac653508dbd96a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:55.995432 6352 consensus_queue.cc:1035] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:55.996178 6133 raft_consensus.cc:1261] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Refusing update from remote peer ed945e883e7344c5aac653508dbd96a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:55.997105 6352 consensus_queue.cc:1035] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:57:56.042554 6371 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 10:57:57.636763 5838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:57.636971 5838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:57.640827 5838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Sending DeleteTablet for 3 replicas of tablet 5be8ef9cd7b74a418a11beca66aca0e5
I20260928 10:57:57.641573 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be8ef9cd7b74a418a11beca66aca0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:57 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40118
I20260928 10:57:57.641661 6253 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be8ef9cd7b74a418a11beca66aca0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:57 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55976
I20260928 10:57:57.641696 6114 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be8ef9cd7b74a418a11beca66aca0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:57 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40608
I20260928 10:57:57.653970 6423 tablet_replica.cc:330] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: stopping tablet replica
I20260928 10:57:57.654723 6423 raft_consensus.cc:2229] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:57.654955 6423 raft_consensus.cc:2258] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:57.656090 6423 ts_tablet_manager.cc:1905] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:57.662277 6425 tablet_replica.cc:330] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: stopping tablet replica
I20260928 10:57:57.662483 6425 raft_consensus.cc:2229] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:57.662528 6424 tablet_replica.cc:330] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: stopping tablet replica
I20260928 10:57:57.662707 6424 raft_consensus.cc:2229] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:57:57.662737 6425 raft_consensus.cc:2258] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:57.663087 6424 raft_consensus.cc:2258] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:57.663905 6423 ts_tablet_manager.cc:1918] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:57:57.664089 6423 log.cc:1196] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/5be8ef9cd7b74a418a11beca66aca0e5
I20260928 10:57:57.664081 6424 ts_tablet_manager.cc:1905] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:57.664553 6423 ts_tablet_manager.cc:1939] T 5be8ef9cd7b74a418a11beca66aca0e5 P a41796cd662d4c6da3e572ec21d2b5ca: Deleting consensus metadata
I20260928 10:57:57.665642 5826 catalog_manager.cc:4841] TS a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295): tablet 5be8ef9cd7b74a418a11beca66aca0e5 (table test-table [id=7e1d094b436e45dd8468aa336d393018]) successfully deleted
I20260928 10:57:57.668746 6425 ts_tablet_manager.cc:1905] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:57.670994 6424 ts_tablet_manager.cc:1918] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:57:57.671113 6424 log.cc:1196] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/5be8ef9cd7b74a418a11beca66aca0e5
I20260928 10:57:57.671485 6424 ts_tablet_manager.cc:1939] T 5be8ef9cd7b74a418a11beca66aca0e5 P ed945e883e7344c5aac653508dbd96a7: Deleting consensus metadata
I20260928 10:57:57.672470 5823 catalog_manager.cc:4841] TS ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889): tablet 5be8ef9cd7b74a418a11beca66aca0e5 (table test-table [id=7e1d094b436e45dd8468aa336d393018]) successfully deleted
I20260928 10:57:57.674774 6425 ts_tablet_manager.cc:1918] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:57:57.674909 6425 log.cc:1196] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/5be8ef9cd7b74a418a11beca66aca0e5
I20260928 10:57:57.675328 6425 ts_tablet_manager.cc:1939] T 5be8ef9cd7b74a418a11beca66aca0e5 P a569538ff0af4513b0e31c050c76e091: Deleting consensus metadata
I20260928 10:57:57.676350 5825 catalog_manager.cc:4841] TS a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255): tablet 5be8ef9cd7b74a418a11beca66aca0e5 (table test-table [id=7e1d094b436e45dd8468aa336d393018]) successfully deleted
Sep 28 10:57:57 dist-test-slave-djjw krb5kdc[5794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:57:57 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:57:57.706956 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:57.687435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.3.252.126:33439)
0928 10:57:57.688120 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:57.688127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:57.688509 (+ 382us) server_negotiation.cc:407] Connection header received
0928 10:57:57.688670 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:57.688674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:57.688741 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:57.688828 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:57.690223 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:57.690992 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:57.692263 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:57.692666 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:57.692772 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:57.702069 (+ 9297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:57.702074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:57.702078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:57.702118 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:57.704499 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:57.705106 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:57.705111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:57.705113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:57.705229 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:57.705575 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:57.705587 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:57.705589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:57.705908 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:57.706120 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:57.706408 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 10:57:57.706632 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":287,"threads_started":1}
I20260928 10:57:57.710209 5838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:57:57.710798 5838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:57.716856 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 27479aa024614c2a87a401a3fbd1a14e (DEFAULT_TABLE table=test-table [id=f1b19cec0a8141eea5f885e95876809c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:57.716943 6114 tablet_service.cc:1467] Processing CreateTablet for tablet 27479aa024614c2a87a401a3fbd1a14e (DEFAULT_TABLE table=test-table [id=f1b19cec0a8141eea5f885e95876809c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:57.717252 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27479aa024614c2a87a401a3fbd1a14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:57.717177 6253 tablet_service.cc:1467] Processing CreateTablet for tablet 27479aa024614c2a87a401a3fbd1a14e (DEFAULT_TABLE table=test-table [id=f1b19cec0a8141eea5f885e95876809c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:57.717356 6114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27479aa024614c2a87a401a3fbd1a14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:57.717527 6253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27479aa024614c2a87a401a3fbd1a14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:57.722154 6434 tablet_bootstrap.cc:492] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: Bootstrap starting.
I20260928 10:57:57.723403 6434 tablet_bootstrap.cc:654] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:57.724390 6435 tablet_bootstrap.cc:492] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: Bootstrap starting.
I20260928 10:57:57.724982 6436 tablet_bootstrap.cc:492] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: Bootstrap starting.
I20260928 10:57:57.725715 6434 tablet_bootstrap.cc:492] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: No bootstrap required, opened a new log
I20260928 10:57:57.725884 6434 ts_tablet_manager.cc:1397] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260928 10:57:57.726562 6435 tablet_bootstrap.cc:654] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:57.726713 6434 raft_consensus.cc:348] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:57.726933 6434 raft_consensus.cc:374] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:57.727006 6434 raft_consensus.cc:729] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41796cd662d4c6da3e572ec21d2b5ca, State: Initialized, Role: FOLLOWER
I20260928 10:57:57.727241 6434 consensus_queue.cc:260] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:57.727784 6434 ts_tablet_manager.cc:1428] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:57:57.728600 6435 tablet_bootstrap.cc:492] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: No bootstrap required, opened a new log
I20260928 10:57:57.728730 6435 ts_tablet_manager.cc:1397] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 10:57:57.728889 6436 tablet_bootstrap.cc:654] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:57.729476 6435 raft_consensus.cc:348] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:57.729701 6435 raft_consensus.cc:374] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:57.729790 6435 raft_consensus.cc:729] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed945e883e7344c5aac653508dbd96a7, State: Initialized, Role: FOLLOWER
I20260928 10:57:57.730020 6435 consensus_queue.cc:260] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:57.730626 6435 ts_tablet_manager.cc:1428] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 10:57:57.731726 6436 tablet_bootstrap.cc:492] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: No bootstrap required, opened a new log
I20260928 10:57:57.731855 6436 ts_tablet_manager.cc:1397] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260928 10:57:57.732580 6436 raft_consensus.cc:348] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:57.732784 6436 raft_consensus.cc:374] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:57.732875 6436 raft_consensus.cc:729] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a569538ff0af4513b0e31c050c76e091, State: Initialized, Role: FOLLOWER
I20260928 10:57:57.733088 6436 consensus_queue.cc:260] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:57.733862 6436 ts_tablet_manager.cc:1428] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 10:57:58.075027 6440 raft_consensus.cc:482] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:58.075278 6440 raft_consensus.cc:504] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.076794 6440 leader_election.cc:288] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889)
I20260928 10:57:58.085896 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.077262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:52303 (local address 127.3.252.67:40889)
0928 10:57:58.078995 (+ 1733us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:58.078999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:58.079019 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:57:58.079086 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:58.079089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:58.079145 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:58.079235 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:58.079858 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.080883 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.083372 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.084666 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.084860 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.084930 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:58.085374 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 10:57:58.085505 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1627,"thread_start_us":1497,"threads_started":1}
I20260928 10:57:58.085899 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.077152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40889 (local address 127.3.252.66:52303)
0928 10:57:58.077790 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:58.077813 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:58.078007 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:58.079282 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:58.079290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:58.079313 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:58.079693 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.079701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.081044 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.081049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.083175 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.083191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.084926 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.084934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.085136 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.085198 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 10:57:58.085317 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 10:57:58.085464 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":55,"thread_start_us":300,"threads_started":1}
I20260928 10:57:58.085976 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.077263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:56363 (local address 127.3.252.65:43295)
0928 10:57:58.078803 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:58.078808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:58.078827 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:57:58.078884 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:58.078888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:58.078948 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:58.079035 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:57:58.079597 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.080665 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.083392 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.084671 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.084846 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.084979 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:58.085484 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 10:57:58.085615 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1419,"thread_start_us":1279,"threads_started":1}
I20260928 10:57:58.086369 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.077152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43295 (local address 127.3.252.66:56363)
0928 10:57:58.078520 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:58.078537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:58.078642 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:58.079077 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:58.079082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:58.079106 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:57:58.079413 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.079425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.080838 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.080843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.083175 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.083191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.085095 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.085100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.085317 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.085343 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:57:58.085444 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 10:57:58.085572 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":154,"thread_start_us":1031,"threads_started":1}
I20260928 10:57:58.086521 6272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27479aa024614c2a87a401a3fbd1a14e" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed945e883e7344c5aac653508dbd96a7" is_pre_election: true
I20260928 10:57:58.087141 6272 raft_consensus.cc:2454] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 0.
I20260928 10:57:58.087414 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27479aa024614c2a87a401a3fbd1a14e" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" is_pre_election: true
I20260928 10:57:58.087700 5994 raft_consensus.cc:2454] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 0.
I20260928 10:57:58.087698 6065 leader_election.cc:302] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a569538ff0af4513b0e31c050c76e091, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:58.088099 6440 raft_consensus.cc:2781] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:58.088240 6440 raft_consensus.cc:482] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:58.088335 6440 raft_consensus.cc:3037] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:58.090523 6440 raft_consensus.cc:504] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.091120 6440 leader_election.cc:288] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 election: Requested vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889)
I20260928 10:57:58.091589 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27479aa024614c2a87a401a3fbd1a14e" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca"
I20260928 10:57:58.091769 5994 raft_consensus.cc:3037] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:58.091722 6272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27479aa024614c2a87a401a3fbd1a14e" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed945e883e7344c5aac653508dbd96a7"
I20260928 10:57:58.091924 6272 raft_consensus.cc:3037] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:58.093704 5994 raft_consensus.cc:2454] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 1.
I20260928 10:57:58.093887 6272 raft_consensus.cc:2454] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 1.
I20260928 10:57:58.094105 6068 leader_election.cc:302] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41796cd662d4c6da3e572ec21d2b5ca, a569538ff0af4513b0e31c050c76e091; no voters:
I20260928 10:57:58.094404 6440 raft_consensus.cc:2781] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:58.094787 6440 raft_consensus.cc:686] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 1 LEADER]: Becoming Leader. State: Replica: a569538ff0af4513b0e31c050c76e091, State: Running, Role: LEADER
I20260928 10:57:58.095166 6440 consensus_queue.cc:237] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.098345 5839 catalog_manager.cc:5495] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 reported cstate change: term changed from 0 to 1, leader changed from <none> to a569538ff0af4513b0e31c050c76e091 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "a569538ff0af4513b0e31c050c76e091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260928 10:57:58.210024 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.203420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.3.252.66:40255)
0928 10:57:58.203939 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:58.203945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:58.203963 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:57:58.204060 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:58.204065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:58.204171 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:58.204340 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:58.204717 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.205979 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.207342 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.207980 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.208109 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.208557 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:58.209300 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:58.209444 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:58.209564 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 10:57:58.209655 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":322,"threads_started":1}
I20260928 10:57:58.225391 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.220217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:60449 (local address 127.3.252.65:43295)
0928 10:57:58.220442 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:58.220448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:58.220656 (+ 208us) server_negotiation.cc:407] Connection header received
0928 10:57:58.220859 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:58.220863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:58.220928 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:58.221003 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:58.221515 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.222395 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.223797 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.224205 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.224312 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.224722 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:58.224825 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:58.224894 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:58.225015 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 10:57:58.225092 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:57:58.225505 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.220028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43295 (local address 127.3.252.66:60449)
0928 10:57:58.220563 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:58.220579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:58.220731 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:58.221045 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:58.221050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:58.221063 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:58.221378 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.221388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.222540 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.222545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.223656 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.223675 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.224368 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.224375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.224494 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.224552 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:58.224911 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:58.224922 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:57:58.224989 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 10:57:58.225134 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":251,"threads_started":1}
I20260928 10:57:58.233385 6272 raft_consensus.cc:1261] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Refusing update from remote peer a569538ff0af4513b0e31c050c76e091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:58.233385 5994 raft_consensus.cc:1261] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Refusing update from remote peer a569538ff0af4513b0e31c050c76e091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:58.234014 6440 consensus_queue.cc:1035] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:58.234328 6445 consensus_queue.cc:1035] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:58.259505 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.254425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:57203 (local address 127.3.252.66:40255)
0928 10:57:58.254585 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:58.254592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:58.254961 (+ 369us) server_negotiation.cc:407] Connection header received
0928 10:57:58.255073 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:58.255076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:58.255137 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:58.255223 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:58.255568 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.256266 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.257893 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.258379 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.258484 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.258843 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:58.258939 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:58.259000 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:58.259183 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 10:57:58.259248 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:57:58.259578 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:58.254382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:40255 (local address 127.3.252.65:57203)
0928 10:57:58.254875 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:58.254892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:58.254995 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:58.255213 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:58.255215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:58.255225 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:57:58.255448 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.255459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.256381 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.256387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.257731 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:58.257743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:58.258547 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:58.258552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:58.258681 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:58.258718 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:57:58.259047 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:57:58.259063 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 10:57:58.259165 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 10:57:58.259284 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":244,"threads_started":1}
I20260928 10:57:58.299233 5838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:58.299489 5838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:57:58.302618 5838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Sending DeleteTablet for 3 replicas of tablet 27479aa024614c2a87a401a3fbd1a14e
I20260928 10:57:58.303468 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet 27479aa024614c2a87a401a3fbd1a14e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:58 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40118
I20260928 10:57:58.303587 6114 tablet_service.cc:1514] Processing DeleteTablet for tablet 27479aa024614c2a87a401a3fbd1a14e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:58 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40608
I20260928 10:57:58.303712 6253 tablet_service.cc:1514] Processing DeleteTablet for tablet 27479aa024614c2a87a401a3fbd1a14e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:57:58 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55976
I20260928 10:57:58.305346 5838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:57:58.306109 5838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:57:58.313122 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 37db85956a494423b4ec33001ab67719 (DEFAULT_TABLE table=test-table [id=00047ff2f0924afab7374f2d4ae76783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:58.313210 6113 tablet_service.cc:1467] Processing CreateTablet for tablet 37db85956a494423b4ec33001ab67719 (DEFAULT_TABLE table=test-table [id=00047ff2f0924afab7374f2d4ae76783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:58.313560 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37db85956a494423b4ec33001ab67719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:58.313560 6113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37db85956a494423b4ec33001ab67719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:58.313498 6252 tablet_service.cc:1467] Processing CreateTablet for tablet 37db85956a494423b4ec33001ab67719 (DEFAULT_TABLE table=test-table [id=00047ff2f0924afab7374f2d4ae76783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:57:58.313913 6252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37db85956a494423b4ec33001ab67719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:58.329542 6473 tablet_bootstrap.cc:492] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: Bootstrap starting.
I20260928 10:57:58.331421 6473 tablet_bootstrap.cc:654] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:58.333281 6470 tablet_replica.cc:330] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: stopping tablet replica
I20260928 10:57:58.333473 6470 raft_consensus.cc:2229] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:58.333701 6470 raft_consensus.cc:2258] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:58.333316 6474 tablet_bootstrap.cc:492] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: Bootstrap starting.
I20260928 10:57:58.334718 6470 ts_tablet_manager.cc:1905] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:58.334282 6471 tablet_replica.cc:330] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: stopping tablet replica
I20260928 10:57:58.335080 6471 raft_consensus.cc:2229] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:57:58.335726 6471 raft_consensus.cc:2258] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:58.337997 6475 tablet_bootstrap.cc:492] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: Bootstrap starting.
I20260928 10:57:58.338497 6471 ts_tablet_manager.cc:1905] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:58.339100 6473 tablet_bootstrap.cc:492] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: No bootstrap required, opened a new log
I20260928 10:57:58.337459 6474 tablet_bootstrap.cc:654] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:58.339241 6473 ts_tablet_manager.cc:1397] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260928 10:57:58.340040 6473 raft_consensus.cc:348] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.340265 6473 raft_consensus.cc:374] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:58.340344 6473 raft_consensus.cc:729] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41796cd662d4c6da3e572ec21d2b5ca, State: Initialized, Role: FOLLOWER
I20260928 10:57:58.340679 6475 tablet_bootstrap.cc:654] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:58.340605 6473 consensus_queue.cc:260] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.344591 6470 ts_tablet_manager.cc:1918] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:58.344720 6470 log.cc:1196] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/27479aa024614c2a87a401a3fbd1a14e
I20260928 10:57:58.345177 6470 ts_tablet_manager.cc:1939] T 27479aa024614c2a87a401a3fbd1a14e P a41796cd662d4c6da3e572ec21d2b5ca: Deleting consensus metadata
I20260928 10:57:58.347289 6472 tablet_replica.cc:330] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: stopping tablet replica
I20260928 10:57:58.347496 6472 raft_consensus.cc:2229] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:57:58.347771 6472 raft_consensus.cc:2258] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:57:58.348486 6471 ts_tablet_manager.cc:1918] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:58.348657 6473 ts_tablet_manager.cc:1428] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260928 10:57:58.349047 6475 tablet_bootstrap.cc:492] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: No bootstrap required, opened a new log
I20260928 10:57:58.349205 6475 ts_tablet_manager.cc:1397] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260928 10:57:58.349391 6471 log.cc:1196] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/27479aa024614c2a87a401a3fbd1a14e
I20260928 10:57:58.350055 6471 ts_tablet_manager.cc:1939] T 27479aa024614c2a87a401a3fbd1a14e P a569538ff0af4513b0e31c050c76e091: Deleting consensus metadata
I20260928 10:57:58.349972 6475 raft_consensus.cc:348] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.350194 6475 raft_consensus.cc:374] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:58.350289 6475 raft_consensus.cc:729] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed945e883e7344c5aac653508dbd96a7, State: Initialized, Role: FOLLOWER
I20260928 10:57:58.350415 6472 ts_tablet_manager.cc:1905] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:57:58.350554 6475 consensus_queue.cc:260] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.351075 6475 ts_tablet_manager.cc:1428] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:57:58.351574 5826 catalog_manager.cc:4841] TS a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295): tablet 27479aa024614c2a87a401a3fbd1a14e (table test-table [id=f1b19cec0a8141eea5f885e95876809c]) successfully deleted
I20260928 10:57:58.352437 5825 catalog_manager.cc:4841] TS a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255): tablet 27479aa024614c2a87a401a3fbd1a14e (table test-table [id=f1b19cec0a8141eea5f885e95876809c]) successfully deleted
I20260928 10:57:58.353670 6474 tablet_bootstrap.cc:492] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: No bootstrap required, opened a new log
I20260928 10:57:58.353812 6474 ts_tablet_manager.cc:1397] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20260928 10:57:58.354679 6474 raft_consensus.cc:348] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.354959 6474 raft_consensus.cc:374] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:58.355068 6474 raft_consensus.cc:729] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a569538ff0af4513b0e31c050c76e091, State: Initialized, Role: FOLLOWER
I20260928 10:57:58.355646 6474 consensus_queue.cc:260] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.356285 6474 ts_tablet_manager.cc:1428] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 10:57:58.358974 6472 ts_tablet_manager.cc:1918] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:57:58.359099 6472 log.cc:1196] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/27479aa024614c2a87a401a3fbd1a14e
I20260928 10:57:58.359437 6472 ts_tablet_manager.cc:1939] T 27479aa024614c2a87a401a3fbd1a14e P ed945e883e7344c5aac653508dbd96a7: Deleting consensus metadata
I20260928 10:57:58.360532 5823 catalog_manager.cc:4841] TS ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889): tablet 27479aa024614c2a87a401a3fbd1a14e (table test-table [id=f1b19cec0a8141eea5f885e95876809c]) successfully deleted
I20260928 10:57:58.449765 6464 raft_consensus.cc:482] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:57:58.450011 6464 raft_consensus.cc:504] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.450861 6464 leader_election.cc:288] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889)
I20260928 10:57:58.451370 6272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db85956a494423b4ec33001ab67719" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed945e883e7344c5aac653508dbd96a7" is_pre_election: true
I20260928 10:57:58.451375 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db85956a494423b4ec33001ab67719" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" is_pre_election: true
I20260928 10:57:58.451586 6272 raft_consensus.cc:2454] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 0.
I20260928 10:57:58.451617 5994 raft_consensus.cc:2454] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 0.
I20260928 10:57:58.451998 6065 leader_election.cc:302] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a569538ff0af4513b0e31c050c76e091, ed945e883e7344c5aac653508dbd96a7; no voters:
I20260928 10:57:58.452284 6464 raft_consensus.cc:2781] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:57:58.452410 6464 raft_consensus.cc:482] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:57:58.452502 6464 raft_consensus.cc:3037] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:58.455675 6464 raft_consensus.cc:504] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.456352 6464 leader_election.cc:288] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 election: Requested vote from peers a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295), ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889)
I20260928 10:57:58.456809 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db85956a494423b4ec33001ab67719" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41796cd662d4c6da3e572ec21d2b5ca"
I20260928 10:57:58.456838 6272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db85956a494423b4ec33001ab67719" candidate_uuid: "a569538ff0af4513b0e31c050c76e091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed945e883e7344c5aac653508dbd96a7"
I20260928 10:57:58.456995 5994 raft_consensus.cc:3037] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:58.457029 6272 raft_consensus.cc:3037] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:58.458988 5994 raft_consensus.cc:2454] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 1.
I20260928 10:57:58.459636 6068 leader_election.cc:302] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41796cd662d4c6da3e572ec21d2b5ca, a569538ff0af4513b0e31c050c76e091; no voters:
I20260928 10:57:58.459851 6272 raft_consensus.cc:2454] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a569538ff0af4513b0e31c050c76e091 in term 1.
I20260928 10:57:58.459888 6464 raft_consensus.cc:2781] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:58.460072 6464 raft_consensus.cc:686] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 1 LEADER]: Becoming Leader. State: Replica: a569538ff0af4513b0e31c050c76e091, State: Running, Role: LEADER
I20260928 10:57:58.460377 6464 consensus_queue.cc:237] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } }
I20260928 10:57:58.463256 5839 catalog_manager.cc:5495] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 reported cstate change: term changed from 0 to 1, leader changed from <none> to a569538ff0af4513b0e31c050c76e091 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "a569538ff0af4513b0e31c050c76e091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a569538ff0af4513b0e31c050c76e091" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 40255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260928 10:57:58.495846 6272 raft_consensus.cc:1261] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Refusing update from remote peer a569538ff0af4513b0e31c050c76e091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:58.495970 5994 raft_consensus.cc:1261] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Refusing update from remote peer a569538ff0af4513b0e31c050c76e091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:58.496706 6463 consensus_queue.cc:1035] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41796cd662d4c6da3e572ec21d2b5ca" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:58.497053 6464 consensus_queue.cc:1035] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed945e883e7344c5aac653508dbd96a7" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:57:58.518498 6450 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 10:58:00.105983 5839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:00.106240 5839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:00.110078 5839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0abb362af7e4b4c9c7d5733b6d9e659: Sending DeleteTablet for 3 replicas of tablet 37db85956a494423b4ec33001ab67719
I20260928 10:58:00.110774 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet 37db85956a494423b4ec33001ab67719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:00 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40118
I20260928 10:58:00.110881 6114 tablet_service.cc:1514] Processing DeleteTablet for tablet 37db85956a494423b4ec33001ab67719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:00 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40608
I20260928 10:58:00.110922 6253 tablet_service.cc:1514] Processing DeleteTablet for tablet 37db85956a494423b4ec33001ab67719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:00 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55976
I20260928 10:58:00.111582 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5910
I20260928 10:58:00.122012 6504 tablet_replica.cc:330] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: stopping tablet replica
I20260928 10:58:00.122360 6504 raft_consensus.cc:2229] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:00.122828 6504 raft_consensus.cc:2258] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:00.123906 6504 ts_tablet_manager.cc:1905] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:00.128446 6505 tablet_replica.cc:330] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: stopping tablet replica
I20260928 10:58:00.128680 6505 raft_consensus.cc:2229] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:00.128906 6505 raft_consensus.cc:2258] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:00.129586 6503 tablet_replica.cc:330] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: stopping tablet replica
I20260928 10:58:00.129788 6503 raft_consensus.cc:2229] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:00.130025 6503 raft_consensus.cc:2258] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:00.130178 6505 ts_tablet_manager.cc:1905] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:00.131057 6503 ts_tablet_manager.cc:1905] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:00.132285 6504 ts_tablet_manager.cc:1918] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:58:00.132413 6504 log.cc:1196] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/37db85956a494423b4ec33001ab67719
I20260928 10:58:00.132870 6504 ts_tablet_manager.cc:1939] T 37db85956a494423b4ec33001ab67719 P a569538ff0af4513b0e31c050c76e091: Deleting consensus metadata
I20260928 10:58:00.134016 5825 catalog_manager.cc:4841] TS a569538ff0af4513b0e31c050c76e091 (127.3.252.66:40255): tablet 37db85956a494423b4ec33001ab67719 (table test-table [id=00047ff2f0924afab7374f2d4ae76783]) successfully deleted
I20260928 10:58:00.136396 6505 ts_tablet_manager.cc:1918] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:58:00.136534 6505 log.cc:1196] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/37db85956a494423b4ec33001ab67719
I20260928 10:58:00.136940 6505 ts_tablet_manager.cc:1939] T 37db85956a494423b4ec33001ab67719 P ed945e883e7344c5aac653508dbd96a7: Deleting consensus metadata
I20260928 10:58:00.137601 6503 ts_tablet_manager.cc:1918] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:58:00.137732 6503 log.cc:1196] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/37db85956a494423b4ec33001ab67719
I20260928 10:58:00.137995 5823 catalog_manager.cc:4841] TS ed945e883e7344c5aac653508dbd96a7 (127.3.252.67:40889): tablet 37db85956a494423b4ec33001ab67719 (table test-table [id=00047ff2f0924afab7374f2d4ae76783]) successfully deleted
I20260928 10:58:00.138172 6503 ts_tablet_manager.cc:1939] T 37db85956a494423b4ec33001ab67719 P a41796cd662d4c6da3e572ec21d2b5ca: Deleting consensus metadata
I20260928 10:58:00.139029 5826 catalog_manager.cc:4841] TS a41796cd662d4c6da3e572ec21d2b5ca (127.3.252.65:43295): tablet 37db85956a494423b4ec33001ab67719 (table test-table [id=00047ff2f0924afab7374f2d4ae76783]) successfully deleted
Sep 28 10:58:00 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:00.148981 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:00.113227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.3.252.65:43295)
0928 10:58:00.134316 (+ 21089us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:00.134323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:00.134796 (+ 473us) server_negotiation.cc:407] Connection header received
0928 10:58:00.134967 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:00.134973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:00.135051 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:00.135147 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:00.136442 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:00.137327 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:00.138800 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:00.139411 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:00.139530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:00.143719 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:00.143735 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:00.143751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:00.143787 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:00.146354 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:00.146997 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:00.147007 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:00.147010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:00.147152 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:00.147497 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:00.147503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:00.147506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:00.147860 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:00.148130 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:00.148429 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 10:58:00.148651 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20947,"thread_start_us":20999,"threads_started":1}
W20260928 10:58:00.149560 6040 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40124
W20260928 10:58:00.150161 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5910: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:58:00.150357 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 5910
I20260928 10:58:00.168700 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6049
Sep 28 10:58:00 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:58:00.183895 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:00.169441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.3.252.66:40255)
0928 10:58:00.169933 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:00.169938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:00.169952 (+ 14us) server_negotiation.cc:407] Connection header received
0928 10:58:00.170004 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:00.170007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:00.170062 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:00.170153 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:00.171764 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:00.172494 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:00.174311 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:00.174729 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:00.174829 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:00.178556 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:00.178569 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:00.178586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:00.178617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:00.181138 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:00.181778 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:00.181785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:00.181787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:00.182038 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:00.182427 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:00.182431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:00.182433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:00.182796 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:00.183026 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:00.183313 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 10:58:00.183550 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
W20260928 10:58:00.184595 6179 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40628
W20260928 10:58:00.185210 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:58:00.185349 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 6049
I20260928 10:58:00.203720 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6188
Sep 28 10:58:00 dist-test-slave-djjw krb5kdc[5794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:58:00.218569 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:00.204425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.3.252.67:40889)
0928 10:58:00.204882 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:00.204887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:00.204904 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:58:00.204957 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:00.204961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:00.205019 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:00.205111 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:00.206342 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:00.207054 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:00.208318 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:00.208734 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:00.208834 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:00.213051 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:00.213067 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:00.213084 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:00.213119 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:00.215930 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:00.216644 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:00.216651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:00.216654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:00.216786 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:00.217178 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:00.217182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:00.217184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:00.217528 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:00.217776 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:00.218084 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 10:58:00.218307 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
W20260928 10:58:00.219225 6318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55988
W20260928 10:58:00.219799 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6188: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:58:00.219973 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 6188
I20260928 10:58:00.237618 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 5807
I20260928 10:58:00.238910 5879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:00.351902 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 5807
2026-09-28T10:58:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8320 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:00 dist-test-slave-djjw krb5kdc[6513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:00 dist-test-slave-djjw krb5kdc[6513](info): set up 2 sockets
Sep 28 10:58:00 dist-test-slave-djjw krb5kdc[6513](info): commencing operation
krb5kdc: starting...
W20260928 10:58:01.350135 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:01 dist-test-slave-djjw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:01.529313 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38693
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:38693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:01.842267 6526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:01.842653 6526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:01.842800 6526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:01.850051 6526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:01.850157 6526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:01.850205 6526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:01.850278 6526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:01.850332 6526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:01.860133 6526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:38693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:01.861426 6526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:01.862713 6526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:01.874142 6531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:01.874208 6534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:01.874246 6532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:01.874817 6526 server_base.cc:1034] running on GCE node
I20260928 10:58:01.875573 6526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:01.876957 6526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:01.878152 6526 hybrid_clock.cc:648] HybridClock initialized: now 1790593081878104 us; error 35 us; skew 500 ppm
I20260928 10:58:01.878494 6526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:01 dist-test-slave-djjw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:01.881582 6526 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:01.882941 6526 webserver.cc:460] Webserver started at http://127.3.252.126:42383/ using document root <none> and password file <none>
I20260928 10:58:01.883750 6526 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:01.883877 6526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:01.884160 6526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:01.887060 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "254ea3f2fc72457baa838b12f942f645"
format_stamp: "Formatted at 2026-09-28 10:58:01 on dist-test-slave-djjw"
server_key: "2d011e8e2ed21f99c53923404175782f"
server_key_iv: "12fe109dcb55cc5e4cdf0e2736ba30e6"
server_key_version: "encryptionkey@0"
I20260928 10:58:01.887789 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "254ea3f2fc72457baa838b12f942f645"
format_stamp: "Formatted at 2026-09-28 10:58:01 on dist-test-slave-djjw"
server_key: "2d011e8e2ed21f99c53923404175782f"
server_key_iv: "12fe109dcb55cc5e4cdf0e2736ba30e6"
server_key_version: "encryptionkey@0"
I20260928 10:58:01.892620 6526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 10:58:01.895816 6541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:01.897091 6526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:01.897250 6526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "254ea3f2fc72457baa838b12f942f645"
format_stamp: "Formatted at 2026-09-28 10:58:01 on dist-test-slave-djjw"
server_key: "2d011e8e2ed21f99c53923404175782f"
server_key_iv: "12fe109dcb55cc5e4cdf0e2736ba30e6"
server_key_version: "encryptionkey@0"
I20260928 10:58:01.897418 6526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:01.933743 6526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:01.937780 6526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:01.938061 6526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:01.954329 6526 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:38693
I20260928 10:58:01.954347 6592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:38693 every 8 connection(s)
I20260928 10:58:01.955700 6526 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:01.956596 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 6526
I20260928 10:58:01.956815 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:01.957468 4081 external_mini_cluster.cc:1428] Setting key 072b34a404f835b3ef13096a6b5f5205
I20260928 10:58:01.959923 6593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:58:01 dist-test-slave-djjw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:01.972687 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645: Bootstrap starting.
I20260928 10:58:01.976513 6593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:01.977686 6593 log.cc:824] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:01.980929 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645: No bootstrap required, opened a new log
I20260928 10:58:01.981380 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:01.959519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.3.252.126:38693)
0928 10:58:01.960287 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:01.960302 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:01.960353 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:58:01.962570 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:01.962599 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:01.963156 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:01.963621 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:01.964887 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:01.966934 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:01.968225 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:01.969214 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:01.969401 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:01.973442 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:01.973453 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:01.973471 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:01.973514 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:01.977136 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:01.977788 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:01.977797 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:01.977805 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:01.978031 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:01.978430 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:01.978433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:01.978436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:01.979011 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:01.979294 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:01.979662 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 10:58:01.980092 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":310,"threads_started":1}
I20260928 10:58:01.988816 6593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ea3f2fc72457baa838b12f942f645" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38693 } }
I20260928 10:58:01.989136 6593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:01.989220 6593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254ea3f2fc72457baa838b12f942f645, State: Initialized, Role: FOLLOWER
I20260928 10:58:01.989718 6593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ea3f2fc72457baa838b12f942f645" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38693 } }
I20260928 10:58:01.989946 6593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:01.990046 6593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:01.990175 6593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:01.992241 6593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ea3f2fc72457baa838b12f942f645" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38693 } }
I20260928 10:58:01.992705 6593 leader_election.cc:302] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254ea3f2fc72457baa838b12f942f645; no voters:
I20260928 10:58:01.993207 6593 leader_election.cc:288] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:01.993316 6598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:01.993908 6598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [term 1 LEADER]: Becoming Leader. State: Replica: 254ea3f2fc72457baa838b12f942f645, State: Running, Role: LEADER
I20260928 10:58:01.994382 6598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ea3f2fc72457baa838b12f942f645" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38693 } }
I20260928 10:58:01.995033 6593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:01.997037 6599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "254ea3f2fc72457baa838b12f942f645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ea3f2fc72457baa838b12f942f645" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38693 } } }
I20260928 10:58:01.997318 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254ea3f2fc72457baa838b12f942f645. Latest consensus state: current_term: 1 leader_uuid: "254ea3f2fc72457baa838b12f942f645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ea3f2fc72457baa838b12f942f645" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38693 } } }
I20260928 10:58:01.997490 6599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:01.997561 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:01.998907 6606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:02.003329 6606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:02.010696 6606 catalog_manager.cc:1347] Generated new cluster ID: 050340a9580e4045ac440ff180312b6a
I20260928 10:58:02.010802 6606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:02.019220 6606 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:02.020946 6606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:02.028396 6606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 254ea3f2fc72457baa838b12f942f645: Generated new TSK 0
I20260928 10:58:02.029178 6606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:58:02.083499 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38693
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:02.404819 6619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:02.405117 6619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:02.405254 6619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:02.412518 6619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:02.412659 6619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:02.412812 6619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:02.422806 6619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:02.424134 6619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:02.425477 6619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:02.438854 6624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:02.438829 6627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:02.439172 6625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:02.440097 6619 server_base.cc:1034] running on GCE node
I20260928 10:58:02.440683 6619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:02.441690 6619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:02.442879 6619 hybrid_clock.cc:648] HybridClock initialized: now 1790593082442825 us; error 50 us; skew 500 ppm
I20260928 10:58:02.443249 6619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:02 dist-test-slave-djjw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593082, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:02.446377 6619 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:58:02.447860 6619 webserver.cc:460] Webserver started at http://127.3.252.65:40769/ using document root <none> and password file <none>
I20260928 10:58:02.448549 6619 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:02.448661 6619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:02.448953 6619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:02.451864 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "843d6926970b4065ad1938a3082e85ca"
format_stamp: "Formatted at 2026-09-28 10:58:02 on dist-test-slave-djjw"
server_key: "fef6b8c7ee510d08d673c6a5e5c6fd5d"
server_key_iv: "30256b2f9609faad901eebcb60334016"
server_key_version: "encryptionkey@0"
I20260928 10:58:02.452641 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "843d6926970b4065ad1938a3082e85ca"
format_stamp: "Formatted at 2026-09-28 10:58:02 on dist-test-slave-djjw"
server_key: "fef6b8c7ee510d08d673c6a5e5c6fd5d"
server_key_iv: "30256b2f9609faad901eebcb60334016"
server_key_version: "encryptionkey@0"
I20260928 10:58:02.457708 6619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:58:02.461318 6634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:02.462741 6619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:02.462893 6619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "843d6926970b4065ad1938a3082e85ca"
format_stamp: "Formatted at 2026-09-28 10:58:02 on dist-test-slave-djjw"
server_key: "fef6b8c7ee510d08d673c6a5e5c6fd5d"
server_key_iv: "30256b2f9609faad901eebcb60334016"
server_key_version: "encryptionkey@0"
I20260928 10:58:02.463088 6619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:02.480652 6619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:02.484937 6619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:02.485208 6619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:02.486192 6619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:02.487512 6619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:02.487598 6619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:02.487718 6619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:02.487769 6619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:02.517174 6619 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:42035
I20260928 10:58:02.517227 6746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:42035 every 8 connection(s)
I20260928 10:58:02.518453 6619 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:02.522819 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 6619
I20260928 10:58:02.523036 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:02.523797 4081 external_mini_cluster.cc:1428] Setting key d4dc92edc47b2722fc59ec8fcfecd777
Sep 28 10:58:02 dist-test-slave-djjw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593082, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:02.539304 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:02.520916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:60277 (local address 127.3.252.126:38693)
0928 10:58:02.521317 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:02.521322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:02.522249 (+ 927us) server_negotiation.cc:407] Connection header received
0928 10:58:02.524631 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:02.524636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:02.524722 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:02.524812 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:02.526506 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:02.527207 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:02.529258 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:02.529665 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:02.529775 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:02.533323 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:02.533330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:02.533334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:02.533373 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:02.536235 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:02.536909 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:02.536914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:02.536918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:02.537055 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:02.537440 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:02.537444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:02.537447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:02.537791 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:02.537980 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:02.538714 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 10:58:02.538937 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":212,"threads_started":1}
I20260928 10:58:02.540079 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:02.521387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38693 (local address 127.3.252.65:60277)
0928 10:58:02.522100 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:02.522137 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:02.524354 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:02.525054 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:02.525062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:02.525560 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:02.526343 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:02.526367 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:02.527374 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:02.527378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:02.529112 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:02.529125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:02.529916 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:02.529923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:02.530217 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:02.530876 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:02.530909 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:02.533094 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:02.536373 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:02.536381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:02.536395 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:02.536792 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:02.537144 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:02.537147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:02.537149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:02.537337 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:02.537955 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:02.537963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:02.538338 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 10:58:02.538565 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 10:58:02.538958 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 10:58:02.541584 6747 heartbeater.cc:347] Connected to a master server at 127.3.252.126:38693
I20260928 10:58:02.541922 6747 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:02.542676 6747 heartbeater.cc:511] Master 127.3.252.126:38693 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
I20260928 10:58:02.544878 6558 ts_manager.cc:194] Registered new tserver with Master: 843d6926970b4065ad1938a3082e85ca (127.3.252.65:42035)
I20260928 10:58:02.546825 6558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:60277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:58:02.567019 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38693
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:02.877604 6753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:02.877938 6753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:02.878077 6753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:02.885272 6753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:02.885393 6753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:02.885550 6753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:02.895257 6753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:02.896582 6753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:02.897887 6753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:02.910712 6758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:02.910672 6761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:02.910739 6759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:02.911429 6753 server_base.cc:1034] running on GCE node
I20260928 10:58:02.912000 6753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:02.912873 6753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:02.914049 6753 hybrid_clock.cc:648] HybridClock initialized: now 1790593082913999 us; error 45 us; skew 500 ppm
I20260928 10:58:02.914337 6753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:02 dist-test-slave-djjw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593082, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:02.920711 6753 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:58:02.922298 6753 webserver.cc:460] Webserver started at http://127.3.252.66:46281/ using document root <none> and password file <none>
I20260928 10:58:02.922966 6753 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:02.923081 6753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:02.923372 6753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:02.926282 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "e92354e803d246c6aa996212d1c459d6"
format_stamp: "Formatted at 2026-09-28 10:58:02 on dist-test-slave-djjw"
server_key: "6bedfa7fa0f13b7ea8c7c7af9dd122ae"
server_key_iv: "ead5b92109d900dba8dc815089a22eee"
server_key_version: "encryptionkey@0"
I20260928 10:58:02.927034 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "e92354e803d246c6aa996212d1c459d6"
format_stamp: "Formatted at 2026-09-28 10:58:02 on dist-test-slave-djjw"
server_key: "6bedfa7fa0f13b7ea8c7c7af9dd122ae"
server_key_iv: "ead5b92109d900dba8dc815089a22eee"
server_key_version: "encryptionkey@0"
I20260928 10:58:02.932140 6753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 10:58:02.935540 6768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:02.936857 6753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:58:02.937004 6753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "e92354e803d246c6aa996212d1c459d6"
format_stamp: "Formatted at 2026-09-28 10:58:02 on dist-test-slave-djjw"
server_key: "6bedfa7fa0f13b7ea8c7c7af9dd122ae"
server_key_iv: "ead5b92109d900dba8dc815089a22eee"
server_key_version: "encryptionkey@0"
I20260928 10:58:02.937181 6753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:02.950362 6753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:02.954519 6753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:02.954793 6753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:02.955791 6753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:02.957103 6753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:02.957180 6753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:02.957293 6753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:02.957343 6753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:02.987828 6753 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:34305
I20260928 10:58:02.987864 6880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:34305 every 8 connection(s)
I20260928 10:58:02.989027 6753 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:02.993356 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 6753
I20260928 10:58:02.993513 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:02.993993 4081 external_mini_cluster.cc:1428] Setting key 41c7d0558adb115482eded85b7fb0884
Sep 28 10:58:02 dist-test-slave-djjw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593082, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:03.008510 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:02.991502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:37193 (local address 127.3.252.126:38693)
0928 10:58:02.991671 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:02.991676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:02.992844 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 10:58:02.994764 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:02.994769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:02.994835 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:02.994911 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:02.996709 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:02.997737 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:02.999323 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:02.999914 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:03.000025 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:03.003366 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:03.003372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:03.003375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:03.003405 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:03.005697 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:03.006376 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:03.006380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:03.006382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:03.006497 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:03.006938 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:03.006942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:03.006944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:03.007188 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:03.007337 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:03.007986 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 10:58:03.008196 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:58:03.009440 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:02.992013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38693 (local address 127.3.252.66:37193)
0928 10:58:02.992679 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:02.992725 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:02.994426 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:02.995110 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:02.995119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:02.995832 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:02.996545 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:02.996564 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:02.997911 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:02.997915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:02.999188 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:02.999208 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:03.000103 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:03.000109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:03.000332 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:03.001003 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:03.001033 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:03.003164 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:03.005818 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:03.005850 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:03.005867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:03.006262 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:03.006594 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:03.006598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:03.006602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:03.006827 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:03.007303 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:03.007311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:03.007632 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 10:58:03.007865 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 10:58:03.008382 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":243,"threads_started":1}
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 10:58:03.010789 6881 heartbeater.cc:347] Connected to a master server at 127.3.252.126:38693
I20260928 10:58:03.011080 6881 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:03.011786 6881 heartbeater.cc:511] Master 127.3.252.126:38693 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
I20260928 10:58:03.013237 6558 ts_manager.cc:194] Registered new tserver with Master: e92354e803d246c6aa996212d1c459d6 (127.3.252.66:34305)
I20260928 10:58:03.014081 6558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:37193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:58:03.033771 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38693
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:03.345865 6886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:03.346138 6886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:03.346230 6886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:03.353365 6886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:03.353480 6886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:03.353590 6886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:03.363260 6886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:03.364497 6886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:03.365751 6886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:03.378636 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:03.378741 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:03.378712 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:03.379179 6886 server_base.cc:1034] running on GCE node
I20260928 10:58:03.379839 6886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:03.380769 6886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:03.381940 6886 hybrid_clock.cc:648] HybridClock initialized: now 1790593083381875 us; error 53 us; skew 500 ppm
I20260928 10:58:03.382259 6886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:03 dist-test-slave-djjw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:03.385181 6886 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:58:03.386643 6886 webserver.cc:460] Webserver started at http://127.3.252.67:40625/ using document root <none> and password file <none>
I20260928 10:58:03.387292 6886 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:03.387383 6886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:03.387626 6886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:03.390547 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "2b6b08c2941446fc835a3da9d0c4912e"
format_stamp: "Formatted at 2026-09-28 10:58:03 on dist-test-slave-djjw"
server_key: "5e1dabc946484588c273e043f6a8788c"
server_key_iv: "f4aa6fea8c7be758ff3772778670e5da"
server_key_version: "encryptionkey@0"
I20260928 10:58:03.391296 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "2b6b08c2941446fc835a3da9d0c4912e"
format_stamp: "Formatted at 2026-09-28 10:58:03 on dist-test-slave-djjw"
server_key: "5e1dabc946484588c273e043f6a8788c"
server_key_iv: "f4aa6fea8c7be758ff3772778670e5da"
server_key_version: "encryptionkey@0"
I20260928 10:58:03.396363 6886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:03.399863 6901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:03.401245 6886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:03.401408 6886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "2b6b08c2941446fc835a3da9d0c4912e"
format_stamp: "Formatted at 2026-09-28 10:58:03 on dist-test-slave-djjw"
server_key: "5e1dabc946484588c273e043f6a8788c"
server_key_iv: "f4aa6fea8c7be758ff3772778670e5da"
server_key_version: "encryptionkey@0"
I20260928 10:58:03.401583 6886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:03.415647 6886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:03.419956 6886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:03.420238 6886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:03.421186 6886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:03.422565 6886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:03.422634 6886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:03.422729 6886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:03.422787 6886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:03.450985 6886 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:44175
I20260928 10:58:03.451030 7013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:44175 every 8 connection(s)
I20260928 10:58:03.452189 6886 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:03.461055 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 6886
I20260928 10:58:03.461237 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:03.461643 4081 external_mini_cluster.cc:1428] Setting key 743781e36c626fa2e859ca69dc8252a6
Sep 28 10:58:03 dist-test-slave-djjw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:03.471336 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:03.454686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:49529 (local address 127.3.252.126:38693)
0928 10:58:03.454897 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:03.454903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:03.456031 (+ 1128us) server_negotiation.cc:407] Connection header received
0928 10:58:03.457811 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:03.457816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:03.457921 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:03.458007 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:03.459502 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:03.460230 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:03.461952 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:03.462433 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:03.462563 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:03.466215 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:03.466219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:03.466230 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:03.466262 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:03.468627 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:03.469335 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:03.469339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:03.469341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:03.469446 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:03.469811 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:03.469814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:03.469817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:03.470118 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:03.470235 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:03.470926 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 10:58:03.471081 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:58:03.472412 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:03.455175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38693 (local address 127.3.252.67:49529)
0928 10:58:03.455870 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:03.455915 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:03.457537 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:03.458236 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:03.458245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:03.458753 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:03.459342 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:03.459359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:03.460389 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:03.460393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:03.461794 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:03.461807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:03.462653 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:03.462661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:03.462940 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:03.463652 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:03.463682 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:03.466033 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:03.468762 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:03.468769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:03.468784 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:03.469238 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:03.469520 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:03.469522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:03.469525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:03.469706 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:03.470245 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:03.470251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:03.470633 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 10:58:03.470873 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 10:58:03.471278 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":273,"threads_started":1}
I20260928 10:58:03.473796 7014 heartbeater.cc:347] Connected to a master server at 127.3.252.126:38693
I20260928 10:58:03.474138 7014 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:03.474814 7014 heartbeater.cc:511] Master 127.3.252.126:38693 requested a full tablet report, sending...
I20260928 10:58:03.476235 6558 ts_manager.cc:194] Registered new tserver with Master: 2b6b08c2941446fc835a3da9d0c4912e (127.3.252.67:44175)
I20260928 10:58:03.477133 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:58:03.477278 6558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:49529
W20260928 10:58:03.505203 6749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:03.502895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.3.252.126:38693)
0928 10:58:03.503073 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:03.503078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:03.503518 (+ 440us) server_negotiation.cc:407] Connection header received
0928 10:58:03.503683 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:03.503687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:03.503750 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:03.503844 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:03.504972 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53310: BlockingRecv error: recv got EOF from 127.0.0.1:53310 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:58:03.508152 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6619
W20260928 10:58:03.510607 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6619: Not authorized: Client connection negotiation failed: client connection to 127.3.252.65:42035: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:58:03.510772 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 6619
I20260928 10:58:03.523072 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6753
W20260928 10:58:03.525651 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6753: Not authorized: Client connection negotiation failed: client connection to 127.3.252.66:34305: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:58:03.525804 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 6753
I20260928 10:58:03.537858 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6886
W20260928 10:58:03.540810 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6886: Not authorized: Client connection negotiation failed: client connection to 127.3.252.67:44175: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:58:03.540972 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 6886
I20260928 10:58:03.553300 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 6526
I20260928 10:58:03.554481 6588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:03.654177 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 6526
2026-09-28T10:58:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3299 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:03 dist-test-slave-djjw krb5kdc[7031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:03 dist-test-slave-djjw krb5kdc[7031](info): set up 2 sockets
Sep 28 10:58:03 dist-test-slave-djjw krb5kdc[7031](info): commencing operation
krb5kdc: starting...
W20260928 10:58:04.658195 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.005s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:04 dist-test-slave-djjw krb5kdc[7031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:04.847718 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:32777
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:05.163559 7044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:05.163852 7044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:05.163976 7044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:05.171270 7044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:05.171387 7044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:05.171443 7044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:05.171521 7044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:05.171583 7044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:05.181526 7044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:05.182854 7044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:05.184165 7044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:05.195611 7052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:05.195698 7050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:05.195649 7049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:05.202088 7044 server_base.cc:1034] running on GCE node
I20260928 10:58:05.202947 7044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:05.204471 7044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:05.205673 7044 hybrid_clock.cc:648] HybridClock initialized: now 1790593085205631 us; error 40 us; skew 500 ppm
I20260928 10:58:05.206048 7044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:05 dist-test-slave-djjw krb5kdc[7031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593085, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:05.209014 7044 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:05.210661 7044 webserver.cc:460] Webserver started at http://127.3.252.126:38817/ using document root <none> and password file <none>
I20260928 10:58:05.211364 7044 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:05.211457 7044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:05.211710 7044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:05.214730 7044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "d210727de2d54cd99389c9f7f585b990"
format_stamp: "Formatted at 2026-09-28 10:58:05 on dist-test-slave-djjw"
server_key: "2b8313a7a6a9b8e22d6546924266adbf"
server_key_iv: "2114f17d4ee18fd6cf625f6fdb990efa"
server_key_version: "encryptionkey@0"
I20260928 10:58:05.215453 7044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "d210727de2d54cd99389c9f7f585b990"
format_stamp: "Formatted at 2026-09-28 10:58:05 on dist-test-slave-djjw"
server_key: "2b8313a7a6a9b8e22d6546924266adbf"
server_key_iv: "2114f17d4ee18fd6cf625f6fdb990efa"
server_key_version: "encryptionkey@0"
I20260928 10:58:05.220556 7044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:05.224263 7059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:05.225667 7044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:05.225888 7044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "d210727de2d54cd99389c9f7f585b990"
format_stamp: "Formatted at 2026-09-28 10:58:05 on dist-test-slave-djjw"
server_key: "2b8313a7a6a9b8e22d6546924266adbf"
server_key_iv: "2114f17d4ee18fd6cf625f6fdb990efa"
server_key_version: "encryptionkey@0"
I20260928 10:58:05.226073 7044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:05.250145 7044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:05.254424 7044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:05.254705 7044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:05.272135 7044 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:32777
I20260928 10:58:05.272157 7110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:32777 every 8 connection(s)
I20260928 10:58:05.273494 7044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:05.275322 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7044
I20260928 10:58:05.275517 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:05.276311 4081 external_mini_cluster.cc:1428] Setting key 01a9398d8c8392c8074f6cb8684c8795
I20260928 10:58:05.278008 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:58:05 dist-test-slave-djjw krb5kdc[7031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:05.291579 7111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990: Bootstrap starting.
I20260928 10:58:05.295092 7111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:05.296331 7111 log.cc:824] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:05.299248 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:05.278246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.3.252.126:32777)
0928 10:58:05.279009 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:05.279021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:05.279059 (+ 38us) server_negotiation.cc:407] Connection header received
0928 10:58:05.280617 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:05.280638 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:05.281059 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:05.281406 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:05.283193 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:05.284579 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:05.286258 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:05.286938 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:05.287079 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:05.291838 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:05.291848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:05.291866 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:05.291911 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:05.294875 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:05.295762 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:05.295768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:05.295776 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:05.295947 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:05.296388 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:05.296392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:05.296394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:05.297188 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:05.297492 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:05.297856 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 10:58:05.298281 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":358,"threads_started":1}
I20260928 10:58:05.299400 7111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990: No bootstrap required, opened a new log
I20260928 10:58:05.302969 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:58:05.308554 7111 raft_consensus.cc:348] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d210727de2d54cd99389c9f7f585b990" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32777 } }
I20260928 10:58:05.309002 7111 raft_consensus.cc:374] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:05.309093 7111 raft_consensus.cc:729] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d210727de2d54cd99389c9f7f585b990, State: Initialized, Role: FOLLOWER
I20260928 10:58:05.309692 7111 consensus_queue.cc:260] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d210727de2d54cd99389c9f7f585b990" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32777 } }
I20260928 10:58:05.309958 7111 raft_consensus.cc:388] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:05.310074 7111 raft_consensus.cc:482] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:05.310227 7111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:05.312373 7111 raft_consensus.cc:504] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d210727de2d54cd99389c9f7f585b990" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32777 } }
I20260928 10:58:05.312858 7111 leader_election.cc:302] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d210727de2d54cd99389c9f7f585b990; no voters:
I20260928 10:58:05.313431 7111 leader_election.cc:288] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:05.313531 7117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:05.314596 7117 raft_consensus.cc:686] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [term 1 LEADER]: Becoming Leader. State: Replica: d210727de2d54cd99389c9f7f585b990, State: Running, Role: LEADER
I20260928 10:58:05.315176 7117 consensus_queue.cc:237] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d210727de2d54cd99389c9f7f585b990" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32777 } }
I20260928 10:58:05.315649 7111 sys_catalog.cc:564] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:05.317986 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d210727de2d54cd99389c9f7f585b990. Latest consensus state: current_term: 1 leader_uuid: "d210727de2d54cd99389c9f7f585b990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d210727de2d54cd99389c9f7f585b990" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32777 } } }
I20260928 10:58:05.318342 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:05.319025 7118 sys_catalog.cc:455] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d210727de2d54cd99389c9f7f585b990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d210727de2d54cd99389c9f7f585b990" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32777 } } }
I20260928 10:58:05.319336 7118 sys_catalog.cc:458] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:05.330480 7124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:05.337489 7124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:05.350765 7124 catalog_manager.cc:1347] Generated new cluster ID: f235b0fa4f514ef2bcca6d7fd3769711
I20260928 10:58:05.350909 7124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:05.369678 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:05.359126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.3.252.126:32777)
0928 10:58:05.359305 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:05.359309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:05.362709 (+ 3400us) server_negotiation.cc:407] Connection header received
0928 10:58:05.362907 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:05.362917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:05.363004 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:05.363118 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:05.364228 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:05.365016 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:05.366442 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:05.366898 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:05.367021 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:05.368108 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:05.368113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:05.368116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:05.368810 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:05.368866 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:05.368890 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:05.368988 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:05.369132 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 10:58:05.369238 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260928 10:58:05.372509 7124 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:05.374403 7124 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:05.380560 7124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d210727de2d54cd99389c9f7f585b990: Generated new TSK 0
I20260928 10:58:05.381534 7124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:58:05.398908 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7044
I20260928 10:58:05.400316 7106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:05.490278 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7044
2026-09-28T10:58:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1831 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:05 dist-test-slave-djjw krb5kdc[7145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:05 dist-test-slave-djjw krb5kdc[7145](info): set up 2 sockets
Sep 28 10:58:05 dist-test-slave-djjw krb5kdc[7145](info): commencing operation
krb5kdc: starting...
W20260928 10:58:06.502224 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:06 dist-test-slave-djjw krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:06.681290 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:39185
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:06.988010 7158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:06.988322 7158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:06.988412 7158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:06.995486 7158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:06.995586 7158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:06.995638 7158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:06.995693 7158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:06.995751 7158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:07.005363 7158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:07.006702 7158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:07.007913 7158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:07.019133 7163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:07.019091 7166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:07.019160 7164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:07.020102 7158 server_base.cc:1034] running on GCE node
I20260928 10:58:07.020859 7158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:07.022270 7158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:07.023478 7158 hybrid_clock.cc:648] HybridClock initialized: now 1790593087023436 us; error 33 us; skew 500 ppm
I20260928 10:58:07.023819 7158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:07 dist-test-slave-djjw krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593087, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:07.026795 7158 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:07.028260 7158 webserver.cc:460] Webserver started at http://127.3.252.126:39769/ using document root <none> and password file <none>
I20260928 10:58:07.028888 7158 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:07.028985 7158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:07.029237 7158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:07.032155 7158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb"
format_stamp: "Formatted at 2026-09-28 10:58:07 on dist-test-slave-djjw"
server_key: "bc0c65cc190eadf151d6b852a5d1b54e"
server_key_iv: "2c2abf93c68b2dcd854a72f3dd2b1a31"
server_key_version: "encryptionkey@0"
I20260928 10:58:07.032860 7158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb"
format_stamp: "Formatted at 2026-09-28 10:58:07 on dist-test-slave-djjw"
server_key: "bc0c65cc190eadf151d6b852a5d1b54e"
server_key_iv: "2c2abf93c68b2dcd854a72f3dd2b1a31"
server_key_version: "encryptionkey@0"
I20260928 10:58:07.038211 7158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 10:58:07.041489 7173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:07.042855 7158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:07.042999 7158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb"
format_stamp: "Formatted at 2026-09-28 10:58:07 on dist-test-slave-djjw"
server_key: "bc0c65cc190eadf151d6b852a5d1b54e"
server_key_iv: "2c2abf93c68b2dcd854a72f3dd2b1a31"
server_key_version: "encryptionkey@0"
I20260928 10:58:07.043184 7158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:07.066327 7158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:07.071400 7158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:07.071672 7158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:07.090173 7158 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:39185
I20260928 10:58:07.090197 7224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:39185 every 8 connection(s)
I20260928 10:58:07.091495 7158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:07.095132 7225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:07.098029 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7158
I20260928 10:58:07.098205 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:07.098832 4081 external_mini_cluster.cc:1428] Setting key 96264fe6332487db7bfc92788ffb9f64
I20260928 10:58:07.107764 7225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb: Bootstrap starting.
I20260928 10:58:07.111186 7225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:07.112162 7225 log.cc:824] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:58:07 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:07.115410 7225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb: No bootstrap required, opened a new log
I20260928 10:58:07.122485 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:07.100931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.3.252.126:39185)
0928 10:58:07.101692 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:07.101708 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:07.101757 (+ 49us) server_negotiation.cc:407] Connection header received
0928 10:58:07.103601 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:07.103630 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:07.104166 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:07.104603 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:07.105785 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:07.107653 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:07.108878 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:07.109935 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:07.110112 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:07.114072 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:07.114083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:07.114176 (+ 93us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:07.114230 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:07.117906 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:07.118648 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:07.118656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:07.118666 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:07.118869 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:07.119233 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:07.119237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:07.119240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:07.119924 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:07.120217 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:07.120535 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 10:58:07.121007 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":314,"threads_started":1}
I20260928 10:58:07.124930 7225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39185 } }
I20260928 10:58:07.125272 7225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:07.125384 7225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb, State: Initialized, Role: FOLLOWER
I20260928 10:58:07.125962 7225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39185 } }
I20260928 10:58:07.126174 7225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:07.126288 7225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:07.126436 7225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:07.128539 7225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39185 } }
I20260928 10:58:07.129005 7225 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb; no voters:
I20260928 10:58:07.129518 7225 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:07.129660 7230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:07.130805 7230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [term 1 LEADER]: Becoming Leader. State: Replica: 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb, State: Running, Role: LEADER
I20260928 10:58:07.131317 7230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39185 } }
I20260928 10:58:07.131647 7225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:07.133970 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39185 } } }
I20260928 10:58:07.134330 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:07.134806 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb. Latest consensus state: current_term: 1 leader_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfdfa2ef4cf4fde8aa0dfcf6d0863fb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39185 } } }
I20260928 10:58:07.135057 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:07.135162 7239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:07.145365 7239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:07.154286 7239 catalog_manager.cc:1347] Generated new cluster ID: e56b6c8ef1024864836763c2b261eaa6
I20260928 10:58:07.154417 7239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:07.177759 7239 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:07.179176 7239 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:07.192862 7239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dfdfa2ef4cf4fde8aa0dfcf6d0863fb: Generated new TSK 0
I20260928 10:58:07.193939 7239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:58:07.249423 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39185
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:07.564900 7251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:07.565205 7251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:07.565331 7251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:07.572552 7251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:07.572676 7251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:07.572832 7251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:07.582826 7251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:07.584132 7251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:07.585420 7251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:07.597767 7259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:07.597808 7256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:07.598225 7257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:07.598512 7251 server_base.cc:1034] running on GCE node
I20260928 10:58:07.599105 7251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:07.600029 7251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:07.601235 7251 hybrid_clock.cc:648] HybridClock initialized: now 1790593087601188 us; error 38 us; skew 500 ppm
I20260928 10:58:07.601536 7251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:07 dist-test-slave-djjw krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593087, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:07.604519 7251 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:58:07.605940 7251 webserver.cc:460] Webserver started at http://127.3.252.65:38195/ using document root <none> and password file <none>
I20260928 10:58:07.606570 7251 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:07.606660 7251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:07.606895 7251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:07.610065 7251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "b447a5543c1f473593c06cd71cfe159d"
format_stamp: "Formatted at 2026-09-28 10:58:07 on dist-test-slave-djjw"
server_key: "72900d46bb245bbcb3de9738d50d8fee"
server_key_iv: "f471ed01e0661778053ae6b7a94a6976"
server_key_version: "encryptionkey@0"
I20260928 10:58:07.611019 7251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "b447a5543c1f473593c06cd71cfe159d"
format_stamp: "Formatted at 2026-09-28 10:58:07 on dist-test-slave-djjw"
server_key: "72900d46bb245bbcb3de9738d50d8fee"
server_key_iv: "f471ed01e0661778053ae6b7a94a6976"
server_key_version: "encryptionkey@0"
I20260928 10:58:07.616423 7251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:07.619988 7266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:07.621711 7251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:07.621915 7251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "b447a5543c1f473593c06cd71cfe159d"
format_stamp: "Formatted at 2026-09-28 10:58:07 on dist-test-slave-djjw"
server_key: "72900d46bb245bbcb3de9738d50d8fee"
server_key_iv: "f471ed01e0661778053ae6b7a94a6976"
server_key_version: "encryptionkey@0"
I20260928 10:58:07.622109 7251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:07.638273 7251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:07.642051 7251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:07.642310 7251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:07.643210 7251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:07.644505 7251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:07.644574 7251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:07.644678 7251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:07.644718 7251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:07.674392 7251 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:34817
I20260928 10:58:07.674427 7378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:34817 every 8 connection(s)
I20260928 10:58:07.675627 7251 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:07.678464 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7251
I20260928 10:58:07.678656 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:07.679271 4081 external_mini_cluster.cc:1428] Setting key 58ba276c910e719699f4bd12ff27a5c4
Sep 28 10:58:07 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593087, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:07.695590 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:07.678049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:49953 (local address 127.3.252.126:39185)
0928 10:58:07.678510 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:07.678517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:07.679568 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 10:58:07.681488 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:07.681493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:07.681568 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:07.681679 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:07.683146 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:07.684083 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:07.685564 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:07.686021 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:07.686118 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:07.689859 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:07.689866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:07.689872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:07.689911 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:07.692308 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:07.693237 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:07.693241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:07.693243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:07.693373 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:07.693866 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:07.693871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:07.693874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:07.694137 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:07.694245 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:07.695117 (+ 872us) server_negotiation.cc:299] Negotiation successful
0928 10:58:07.695336 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 10:58:07.696933 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:07.678638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39185 (local address 127.3.252.65:49953)
0928 10:58:07.679410 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:07.679455 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:07.681239 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:07.681887 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:07.681899 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:07.682406 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:07.682977 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:07.682995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:07.684211 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:07.684214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:07.685429 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:07.685442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:07.686246 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:07.686252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:07.686566 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:07.687253 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:07.687308 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:07.689585 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:07.692471 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:07.692480 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:07.692505 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:07.693123 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:07.693479 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:07.693484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:07.693487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:07.693736 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:07.694261 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:07.694269 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:07.694717 (+ 448us) client_negotiation.cc:769] Sending connection context
0928 10:58:07.695047 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 10:58:07.695595 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":291,"threads_started":1}
I20260928 10:58:07.698433 7379 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
I20260928 10:58:07.698745 7379 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:07.699483 7379 heartbeater.cc:511] Master 127.3.252.126:39185 requested a full tablet report, sending...
I20260928 10:58:07.701715 7190 ts_manager.cc:194] Registered new tserver with Master: b447a5543c1f473593c06cd71cfe159d (127.3.252.65:34817)
I20260928 10:58:07.703583 7190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:49953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:58:07.721817 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39185
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:08.034832 7385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:08.035143 7385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:08.035280 7385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:08.042649 7385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:08.042778 7385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:08.042937 7385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:08.052877 7385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:08.054365 7385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:08.055682 7385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:08.068145 7393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:08.068188 7390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:08.068224 7391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:08.068819 7385 server_base.cc:1034] running on GCE node
I20260928 10:58:08.069420 7385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:08.070391 7385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:08.071600 7385 hybrid_clock.cc:648] HybridClock initialized: now 1790593088071546 us; error 44 us; skew 500 ppm
I20260928 10:58:08.071943 7385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:08.077808 7385 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:58:08.079211 7385 webserver.cc:460] Webserver started at http://127.3.252.66:34249/ using document root <none> and password file <none>
I20260928 10:58:08.079885 7385 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:08.079998 7385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:08.080283 7385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:08.083206 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "466dd8460f1f4467b1e1c39da3a76528"
format_stamp: "Formatted at 2026-09-28 10:58:08 on dist-test-slave-djjw"
server_key: "925719dab9df160b21c22e992f06436a"
server_key_iv: "5e257a08d1fbecf5f71a498a39c7e66e"
server_key_version: "encryptionkey@0"
I20260928 10:58:08.083956 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "466dd8460f1f4467b1e1c39da3a76528"
format_stamp: "Formatted at 2026-09-28 10:58:08 on dist-test-slave-djjw"
server_key: "925719dab9df160b21c22e992f06436a"
server_key_iv: "5e257a08d1fbecf5f71a498a39c7e66e"
server_key_version: "encryptionkey@0"
I20260928 10:58:08.088925 7385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 10:58:08.092429 7400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:08.093796 7385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:08.093963 7385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "466dd8460f1f4467b1e1c39da3a76528"
format_stamp: "Formatted at 2026-09-28 10:58:08 on dist-test-slave-djjw"
server_key: "925719dab9df160b21c22e992f06436a"
server_key_iv: "5e257a08d1fbecf5f71a498a39c7e66e"
server_key_version: "encryptionkey@0"
I20260928 10:58:08.094133 7385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:08.112743 7385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:08.116647 7385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:08.116919 7385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:08.117890 7385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:08.119313 7385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:08.119434 7385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:58:08.119560 7385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:08.119617 7385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:08.151203 7385 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:37919
I20260928 10:58:08.151240 7512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:37919 every 8 connection(s)
I20260928 10:58:08.152434 7385 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:08.159991 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7385
I20260928 10:58:08.160128 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:08.160568 4081 external_mini_cluster.cc:1428] Setting key b87d33f093f53c210be804b3052c6940
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:08.171388 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.154849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:51879 (local address 127.3.252.126:39185)
0928 10:58:08.155046 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:08.155052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:08.156197 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 10:58:08.157980 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:08.157984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:08.158052 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:08.158138 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:08.159587 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.160385 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.162070 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.162528 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.162652 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.165862 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:08.165868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:08.165873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:08.165917 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:08.168504 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.169192 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.169198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.169201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.169352 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.169720 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.169724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.169727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.170068 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:08.170211 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:08.170849 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 10:58:08.171070 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:58:08.172246 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.155373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39185 (local address 127.3.252.66:51879)
0928 10:58:08.156040 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:08.156082 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:08.157668 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:08.158370 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:08.158379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:08.158880 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:08.159433 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:08.159450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.160534 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.160538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:08.161934 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:08.161946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.162724 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.162729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:08.162922 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.163545 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:08.163571 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:08.165618 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:08.168626 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:08.168632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:08.168646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:08.169067 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:08.169453 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:08.169456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:08.169458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:08.169628 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:08.170174 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:08.170180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:08.170515 (+ 335us) client_negotiation.cc:769] Sending connection context
0928 10:58:08.170741 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 10:58:08.171130 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":251,"threads_started":1}
I20260928 10:58:08.173698 7513 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39185
I20260928 10:58:08.174041 7513 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:08.174942 7513 heartbeater.cc:511] Master 127.3.252.126:39185 requested a full tablet report, sending...
I20260928 10:58:08.176589 7190 ts_manager.cc:194] Registered new tserver with Master: 466dd8460f1f4467b1e1c39da3a76528 (127.3.252.66:37919)
I20260928 10:58:08.177708 7190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:51879
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:58:08.202893 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39185
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:08.512096 7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:08.512384 7518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:08.512507 7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:08.519609 7518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:08.519726 7518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:08.519881 7518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:08.529551 7518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:08.530892 7518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:08.532202 7518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:08.544703 7523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:08.544627 7526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:08.544656 7524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:08.545382 7518 server_base.cc:1034] running on GCE node
I20260928 10:58:08.546082 7518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:08.547132 7518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:08.548342 7518 hybrid_clock.cc:648] HybridClock initialized: now 1790593088548298 us; error 36 us; skew 500 ppm
I20260928 10:58:08.548635 7518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:08.551713 7518 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:58:08.553135 7518 webserver.cc:460] Webserver started at http://127.3.252.67:44205/ using document root <none> and password file <none>
I20260928 10:58:08.553790 7518 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:08.553936 7518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:08.554214 7518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:08.557117 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "017f13a41de14c9ca8c586329b38eeee"
format_stamp: "Formatted at 2026-09-28 10:58:08 on dist-test-slave-djjw"
server_key: "9c000ec810fc7c1bb90ea9be795e63db"
server_key_iv: "fd4e27d5809cf3a129a5cdb18bb0c30d"
server_key_version: "encryptionkey@0"
I20260928 10:58:08.557924 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "017f13a41de14c9ca8c586329b38eeee"
format_stamp: "Formatted at 2026-09-28 10:58:08 on dist-test-slave-djjw"
server_key: "9c000ec810fc7c1bb90ea9be795e63db"
server_key_iv: "fd4e27d5809cf3a129a5cdb18bb0c30d"
server_key_version: "encryptionkey@0"
I20260928 10:58:08.562947 7518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 10:58:08.566385 7533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:08.567768 7518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 10:58:08.567919 7518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "017f13a41de14c9ca8c586329b38eeee"
format_stamp: "Formatted at 2026-09-28 10:58:08 on dist-test-slave-djjw"
server_key: "9c000ec810fc7c1bb90ea9be795e63db"
server_key_iv: "fd4e27d5809cf3a129a5cdb18bb0c30d"
server_key_version: "encryptionkey@0"
I20260928 10:58:08.568100 7518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:08.587265 7518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:08.591380 7518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:08.591650 7518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:08.592540 7518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:08.593804 7518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:08.593895 7518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:08.594018 7518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:08.594065 7518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:08.623658 7518 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:40103
I20260928 10:58:08.623775 7645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:40103 every 8 connection(s)
I20260928 10:58:08.625033 7518 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:08.629762 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7518
I20260928 10:58:08.629938 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:08.630453 4081 external_mini_cluster.cc:1428] Setting key b62a24e23ad6563193248394537449f1
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:08.644272 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.627702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:57757 (local address 127.3.252.126:39185)
0928 10:58:08.627864 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:08.627868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:08.629047 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 10:58:08.630920 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:08.630925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:08.631009 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:08.631096 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:08.632646 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.633530 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.635075 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.635495 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.635602 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.638968 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:08.638973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:08.638977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:08.639009 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:08.641190 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.641953 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.641957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.641960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.642144 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.642557 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.642560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.642562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.642829 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:08.642955 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:08.643668 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 10:58:08.643905 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:58:08.645330 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.628217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39185 (local address 127.3.252.67:57757)
0928 10:58:08.628896 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:08.628934 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:08.630661 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:08.631353 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:08.631362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:08.631890 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:08.632486 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:08.632504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.633680 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.633684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:08.634939 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:08.634950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.635707 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.635714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:08.635937 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.636563 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:08.636590 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:08.638765 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:08.641333 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:08.641340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:08.641353 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:08.641803 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:08.642261 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:08.642264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:08.642267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:08.642444 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:08.642952 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:08.642960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:08.643307 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 10:58:08.643545 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 10:58:08.644085 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":249,"threads_started":1}
I20260928 10:58:08.646564 7646 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39185
I20260928 10:58:08.646843 7646 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:08.647578 7646 heartbeater.cc:511] Master 127.3.252.126:39185 requested a full tablet report, sending...
I20260928 10:58:08.648881 7190 ts_manager.cc:194] Registered new tserver with Master: 017f13a41de14c9ca8c586329b38eeee (127.3.252.67:40103)
I20260928 10:58:08.649632 7190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:57757
I20260928 10:58:08.657516 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:08.702172 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.688636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.3.252.126:39185)
0928 10:58:08.688886 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:08.688892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:08.689136 (+ 244us) server_negotiation.cc:407] Connection header received
0928 10:58:08.689299 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:08.689303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:08.689370 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:08.689457 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:08.690899 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.691687 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.692959 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.693401 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.693500 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.697160 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:08.697167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:08.697172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:08.697228 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:08.699729 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.700372 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.700376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.700378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.700507 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.700913 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.700918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.700920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.701249 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:08.701453 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:08.701686 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 10:58:08.701887 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260928 10:58:08.702759 7190 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55052
I20260928 10:58:08.704200 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7251
I20260928 10:58:08.706944 7379 heartbeater.cc:503] Master 127.3.252.126:39185 was elected leader, sending a full tablet report...
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:08.719187 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.705211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.3.252.65:34817)
0928 10:58:08.705807 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:08.705814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:08.705861 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:58:08.705923 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:08.705929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:08.706131 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:08.706258 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:08.707406 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.708568 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.709755 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.710363 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.710459 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.714154 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:08.714166 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:08.714182 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:08.714215 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:08.716747 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.717376 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.717382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.717384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.717496 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.717803 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.717806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.717807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.718169 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:08.718405 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:08.718650 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 10:58:08.718843 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":271,"threads_started":1}
W20260928 10:58:08.720376 7374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40436
W20260928 10:58:08.721415 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:58:08.721607 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7251
I20260928 10:58:08.734174 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7385
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:58:08.748940 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.735053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.3.252.66:37919)
0928 10:58:08.735594 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:08.735601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:08.735626 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:58:08.735689 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:08.735696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:08.735898 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:08.736026 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:08.737257 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.738363 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.739621 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.740146 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.740275 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.744013 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:08.744023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:08.744038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:08.744069 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:08.746634 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.747262 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.747269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.747271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.747385 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.747675 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.747679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.747682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.747987 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:08.748204 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:08.748404 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 10:58:08.748594 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":306,"threads_started":1}
W20260928 10:58:08.750213 7508 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52328
W20260928 10:58:08.751287 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7385: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:58:08.751418 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7385
I20260928 10:58:08.764070 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7518
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:58:08.779311 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:08.764974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.3.252.67:40103)
0928 10:58:08.765619 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:08.765626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:08.765655 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:58:08.765714 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:08.765720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:08.765986 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:08.766114 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:08.767319 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.768322 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.769540 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:08.770141 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:08.770239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:08.774306 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:08.774317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:08.774333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:08.774371 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:08.776842 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.777483 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.777489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.777491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.777609 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:08.777914 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:08.777919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:08.777922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:08.778283 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:08.778507 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:08.778804 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 10:58:08.779017 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":340,"threads_started":1}
W20260928 10:58:08.780568 7641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55288
W20260928 10:58:08.781653 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7518: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:58:08.781822 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7518
I20260928 10:58:08.793939 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7158
I20260928 10:58:08.795053 7220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:08.895153 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7158
2026-09-28T10:58:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3409 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7663](info): set up 2 sockets
Sep 28 10:58:08 dist-test-slave-djjw krb5kdc[7663](info): commencing operation
krb5kdc: starting...
W20260928 10:58:09.890156 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:10 dist-test-slave-djjw krb5kdc[7663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:10.073869 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41375
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:10.390200 7676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:10.390509 7676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:10.390626 7676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:10.397969 7676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:10.398085 7676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:10.398137 7676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:10.398212 7676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:10.398262 7676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:10.408255 7676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:41375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:10.409582 7676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:10.410981 7676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:10.422946 7681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:10.422979 7684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:10.423027 7682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:10.423969 7676 server_base.cc:1034] running on GCE node
I20260928 10:58:10.424829 7676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:10.426409 7676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:10.427645 7676 hybrid_clock.cc:648] HybridClock initialized: now 1790593090427595 us; error 38 us; skew 500 ppm
I20260928 10:58:10.428006 7676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:10 dist-test-slave-djjw krb5kdc[7663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593090, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:10.431288 7676 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:10.432855 7676 webserver.cc:460] Webserver started at http://127.3.252.126:43865/ using document root <none> and password file <none>
I20260928 10:58:10.433514 7676 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:10.433616 7676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:10.433929 7676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:10.436885 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "b7bc46b749734f1f91dfe73e47c8d756"
format_stamp: "Formatted at 2026-09-28 10:58:10 on dist-test-slave-djjw"
server_key: "bcd6e1c542e9266bd49326088611edab"
server_key_iv: "210050b14ed868dfe4ffef9baaa905da"
server_key_version: "encryptionkey@0"
I20260928 10:58:10.437613 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "b7bc46b749734f1f91dfe73e47c8d756"
format_stamp: "Formatted at 2026-09-28 10:58:10 on dist-test-slave-djjw"
server_key: "bcd6e1c542e9266bd49326088611edab"
server_key_iv: "210050b14ed868dfe4ffef9baaa905da"
server_key_version: "encryptionkey@0"
I20260928 10:58:10.442823 7676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:58:10.446317 7691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:10.447649 7676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 10:58:10.447819 7676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "b7bc46b749734f1f91dfe73e47c8d756"
format_stamp: "Formatted at 2026-09-28 10:58:10 on dist-test-slave-djjw"
server_key: "bcd6e1c542e9266bd49326088611edab"
server_key_iv: "210050b14ed868dfe4ffef9baaa905da"
server_key_version: "encryptionkey@0"
I20260928 10:58:10.448019 7676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:10.464634 7676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:10.468918 7676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:10.469183 7676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:10.485970 7676 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:41375
I20260928 10:58:10.485977 7742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:41375 every 8 connection(s)
I20260928 10:58:10.487411 7676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:10.491171 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7676
I20260928 10:58:10.491397 7743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:10.491416 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:10.492185 4081 external_mini_cluster.cc:1428] Setting key 96fccbef68c30c41feb90c22ac3bc781
I20260928 10:58:10.505304 7743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756: Bootstrap starting.
Sep 28 10:58:10 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:10.509028 7743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:10.510138 7743 log.cc:824] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:10.513309 7743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756: No bootstrap required, opened a new log
I20260928 10:58:10.515983 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:10.494423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.3.252.126:41375)
0928 10:58:10.495197 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:10.495209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:10.495251 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:58:10.497565 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:10.497594 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:10.498088 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:10.498440 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:10.500205 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:10.501488 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:10.503316 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:10.503978 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:10.504102 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:10.508626 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:10.508633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:10.508648 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:10.508693 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:10.511877 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:10.512816 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:10.512822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:10.512829 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:10.512985 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:10.513381 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:10.513386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:10.513389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:10.513941 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:10.514169 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:10.514574 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 10:58:10.514919 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":363,"threads_started":1}
I20260928 10:58:10.521467 7743 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc46b749734f1f91dfe73e47c8d756" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41375 } }
I20260928 10:58:10.521893 7743 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:10.522002 7743 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bc46b749734f1f91dfe73e47c8d756, State: Initialized, Role: FOLLOWER
I20260928 10:58:10.522552 7743 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc46b749734f1f91dfe73e47c8d756" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41375 } }
I20260928 10:58:10.522763 7743 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:10.522855 7743 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:10.522969 7743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:10.525177 7743 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc46b749734f1f91dfe73e47c8d756" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41375 } }
I20260928 10:58:10.525609 7743 leader_election.cc:302] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7bc46b749734f1f91dfe73e47c8d756; no voters:
I20260928 10:58:10.526244 7743 leader_election.cc:288] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:10.526384 7748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:10.526960 7748 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [term 1 LEADER]: Becoming Leader. State: Replica: b7bc46b749734f1f91dfe73e47c8d756, State: Running, Role: LEADER
I20260928 10:58:10.527599 7748 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc46b749734f1f91dfe73e47c8d756" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41375 } }
I20260928 10:58:10.528419 7743 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:10.530936 7749 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7bc46b749734f1f91dfe73e47c8d756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc46b749734f1f91dfe73e47c8d756" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41375 } } }
I20260928 10:58:10.531028 7750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7bc46b749734f1f91dfe73e47c8d756. Latest consensus state: current_term: 1 leader_uuid: "b7bc46b749734f1f91dfe73e47c8d756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc46b749734f1f91dfe73e47c8d756" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41375 } } }
I20260928 10:58:10.531338 7749 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:10.531368 7750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:10.533028 7757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:10.537637 7757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:10.545909 7757 catalog_manager.cc:1347] Generated new cluster ID: f3bb65c251b341d8a4e1833db413d497
I20260928 10:58:10.546020 7757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:10.569626 7757 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:10.570989 7757 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:10.580359 7757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7bc46b749734f1f91dfe73e47c8d756: Generated new TSK 0
I20260928 10:58:10.581187 7757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:58:10.628695 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41375
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:10.946769 7769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:10.947043 7769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:10.947134 7769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:10.954465 7769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:10.954588 7769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:10.954705 7769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:10.964671 7769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:10.965996 7769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:10.967307 7769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:10.979764 7774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:10.979805 7775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:10.979825 7777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:10.980611 7769 server_base.cc:1034] running on GCE node
I20260928 10:58:10.981200 7769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:10.982158 7769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:10.983364 7769 hybrid_clock.cc:648] HybridClock initialized: now 1790593090983303 us; error 54 us; skew 500 ppm
I20260928 10:58:10.983664 7769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:10 dist-test-slave-djjw krb5kdc[7663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593090, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:10.986649 7769 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:58:10.987982 7769 webserver.cc:460] Webserver started at http://127.3.252.65:36069/ using document root <none> and password file <none>
I20260928 10:58:10.988623 7769 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:10.988718 7769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:10.988963 7769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:10.991792 7769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "d25cdf2b96ed4bc3b2acd9588c4ef8a1"
format_stamp: "Formatted at 2026-09-28 10:58:10 on dist-test-slave-djjw"
server_key: "09f0c27e019187e284f77d2f39ad069e"
server_key_iv: "5c029f20e6ece83967f7be134cb74ed8"
server_key_version: "encryptionkey@0"
I20260928 10:58:10.992507 7769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "d25cdf2b96ed4bc3b2acd9588c4ef8a1"
format_stamp: "Formatted at 2026-09-28 10:58:10 on dist-test-slave-djjw"
server_key: "09f0c27e019187e284f77d2f39ad069e"
server_key_iv: "5c029f20e6ece83967f7be134cb74ed8"
server_key_version: "encryptionkey@0"
I20260928 10:58:10.997581 7769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:58:11.000952 7784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:11.002322 7769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:11.002452 7769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "d25cdf2b96ed4bc3b2acd9588c4ef8a1"
format_stamp: "Formatted at 2026-09-28 10:58:10 on dist-test-slave-djjw"
server_key: "09f0c27e019187e284f77d2f39ad069e"
server_key_iv: "5c029f20e6ece83967f7be134cb74ed8"
server_key_version: "encryptionkey@0"
I20260928 10:58:11.002620 7769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:11.017948 7769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:11.022235 7769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:11.022506 7769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:11.023478 7769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:11.024737 7769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:11.024807 7769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:11.024920 7769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:11.024977 7769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:11.054965 7769 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:40835
I20260928 10:58:11.055004 7896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:40835 every 8 connection(s)
I20260928 10:58:11.056208 7769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:11.056510 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7769
I20260928 10:58:11.056710 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:11.057495 4081 external_mini_cluster.cc:1428] Setting key 23dae8542bbbadc8aedd570513872cb4
Sep 28 10:58:11 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593090, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 10:58:11.078337 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:11.058859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:60451 (local address 127.3.252.126:41375)
0928 10:58:11.059246 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:11.059251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:11.060602 (+ 1351us) server_negotiation.cc:407] Connection header received
0928 10:58:11.063030 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:11.063035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:11.063102 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:11.063215 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:11.065222 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.066020 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.067877 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.068519 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.068654 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:11.072124 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:11.072131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:11.072135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:11.072180 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:11.075400 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:11.076085 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:11.076091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:11.076095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:11.076253 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:11.076646 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:11.076651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:11.076654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:11.077013 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:11.077147 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:11.077858 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 10:58:11.078027 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":208,"threads_started":1}
I20260928 10:58:11.079286 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:11.059605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:41375 (local address 127.3.252.65:60451)
0928 10:58:11.060439 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:11.060486 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:11.062694 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:11.063485 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:11.063497 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:11.064220 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:11.065032 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:11.065057 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.066185 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.066190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:11.067713 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:11.067725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.068716 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.068721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:11.068931 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:11.069579 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:11.069603 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:11.071867 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:11.075538 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:11.075545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:11.075559 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:11.075963 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:11.076364 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:11.076367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:11.076369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:11.076545 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:11.077125 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:11.077130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:11.077480 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 10:58:11.077707 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 10:58:11.078155 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":301,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
I20260928 10:58:11.080875 7897 heartbeater.cc:347] Connected to a master server at 127.3.252.126:41375
I20260928 10:58:11.081184 7897 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:11.081903 7897 heartbeater.cc:511] Master 127.3.252.126:41375 requested a full tablet report, sending...
I20260928 10:58:11.083959 7708 ts_manager.cc:194] Registered new tserver with Master: d25cdf2b96ed4bc3b2acd9588c4ef8a1 (127.3.252.65:40835)
I20260928 10:58:11.085323 7708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:60451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:58:11.106220 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41375
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:11.423077 7903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:11.423386 7903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:11.423491 7903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:11.430730 7903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:11.430848 7903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:11.430970 7903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:11.440865 7903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:11.442159 7903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:11.443506 7903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:11.456624 7908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:11.456684 7911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:11.456672 7909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:11.457309 7903 server_base.cc:1034] running on GCE node
I20260928 10:58:11.457983 7903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:11.458917 7903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:11.460119 7903 hybrid_clock.cc:648] HybridClock initialized: now 1790593091460060 us; error 54 us; skew 500 ppm
I20260928 10:58:11.460431 7903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:11 dist-test-slave-djjw krb5kdc[7663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593091, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:11.463549 7903 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:58:11.465015 7903 webserver.cc:460] Webserver started at http://127.3.252.66:38071/ using document root <none> and password file <none>
I20260928 10:58:11.465687 7903 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:11.465796 7903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:11.466112 7903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:11.468976 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "758559681b3d4e9c9cd355a222a65bcf"
format_stamp: "Formatted at 2026-09-28 10:58:11 on dist-test-slave-djjw"
server_key: "bf59e2ac8ac535f508ecf3b5b1ff2b41"
server_key_iv: "6f2ab233ae4c3aceda83b136601b6d3d"
server_key_version: "encryptionkey@0"
I20260928 10:58:11.469729 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "758559681b3d4e9c9cd355a222a65bcf"
format_stamp: "Formatted at 2026-09-28 10:58:11 on dist-test-slave-djjw"
server_key: "bf59e2ac8ac535f508ecf3b5b1ff2b41"
server_key_iv: "6f2ab233ae4c3aceda83b136601b6d3d"
server_key_version: "encryptionkey@0"
I20260928 10:58:11.475081 7903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:58:11.478607 7918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:11.480079 7903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 10:58:11.480257 7903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "758559681b3d4e9c9cd355a222a65bcf"
format_stamp: "Formatted at 2026-09-28 10:58:11 on dist-test-slave-djjw"
server_key: "bf59e2ac8ac535f508ecf3b5b1ff2b41"
server_key_iv: "6f2ab233ae4c3aceda83b136601b6d3d"
server_key_version: "encryptionkey@0"
I20260928 10:58:11.480477 7903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:11.501569 7903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:11.506814 7903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:11.507128 7903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:11.508121 7903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:11.509475 7903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:11.509545 7903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:11.509631 7903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:11.509672 7903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:11.539054 7903 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:40857
I20260928 10:58:11.539085 8030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:40857 every 8 connection(s)
I20260928 10:58:11.540282 7903 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:11.544366 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 7903
I20260928 10:58:11.544534 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:11.544921 4081 external_mini_cluster.cc:1428] Setting key 9573c886a0ef1fdf22c6d99f9bd5016b
Sep 28 10:58:11 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593091, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:11.559708 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:11.542718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:60051 (local address 127.3.252.126:41375)
0928 10:58:11.542889 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:11.542894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:11.544037 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 10:58:11.545933 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:11.545939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:11.546021 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:11.546112 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:11.547689 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.548389 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.549893 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.550286 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.550392 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:11.553791 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:11.553799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:11.553804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:11.553865 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:11.556659 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:11.557385 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:11.557391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:11.557394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:11.557540 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:11.557996 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:11.558002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:11.558005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:11.558339 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:11.558488 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:11.559124 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 10:58:11.559350 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:58:11.560581 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:11.543198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:41375 (local address 127.3.252.66:60051)
0928 10:58:11.543873 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:11.543916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:11.545618 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:11.546377 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:11.546385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:11.546945 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:11.547519 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:11.547537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.548532 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.548536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:11.549743 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:11.549755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:11.550503 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:11.550510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:11.550718 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:11.551316 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:11.551352 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:11.553567 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:11.556835 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:11.556843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:11.556858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:11.557275 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:11.557652 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:11.557655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:11.557657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:11.557876 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:11.558454 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:11.558460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:11.558782 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 10:58:11.559007 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 10:58:11.559426 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":254,"threads_started":1}
I20260928 10:58:11.562050 8031 heartbeater.cc:347] Connected to a master server at 127.3.252.126:41375
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 10:58:11.562391 8031 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:11.563143 8031 heartbeater.cc:511] Master 127.3.252.126:41375 requested a full tablet report, sending...
I20260928 10:58:11.564574 7708 ts_manager.cc:194] Registered new tserver with Master: 758559681b3d4e9c9cd355a222a65bcf (127.3.252.66:40857)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
I20260928 10:58:11.565448 7708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:60051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:58:11.586740 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41375
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:11.902499 8036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:11.902791 8036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:11.902912 8036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:11.910367 8036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:11.910493 8036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:11.910650 8036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:11.920238 8036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:11.921494 8036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:11.922783 8036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:11.935424 8041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:11.935475 8042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:11.935498 8044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:11.936190 8036 server_base.cc:1034] running on GCE node
I20260928 10:58:11.936743 8036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:11.937595 8036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:11.938786 8036 hybrid_clock.cc:648] HybridClock initialized: now 1790593091938732 us; error 47 us; skew 500 ppm
I20260928 10:58:11.939067 8036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:11 dist-test-slave-djjw krb5kdc[7663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593091, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:11.941985 8036 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:58:11.943420 8036 webserver.cc:460] Webserver started at http://127.3.252.67:40513/ using document root <none> and password file <none>
I20260928 10:58:11.944061 8036 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:11.944159 8036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:11.944429 8036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:11.947273 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "13f2f770a5974db2a74b252cd24b58dd"
format_stamp: "Formatted at 2026-09-28 10:58:11 on dist-test-slave-djjw"
server_key: "cd1b31bcb294d4d6011b9001385023d1"
server_key_iv: "39ac56e94f34e4277a0862dd35fae463"
server_key_version: "encryptionkey@0"
I20260928 10:58:11.948001 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "13f2f770a5974db2a74b252cd24b58dd"
format_stamp: "Formatted at 2026-09-28 10:58:11 on dist-test-slave-djjw"
server_key: "cd1b31bcb294d4d6011b9001385023d1"
server_key_iv: "39ac56e94f34e4277a0862dd35fae463"
server_key_version: "encryptionkey@0"
I20260928 10:58:11.953125 8036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:58:11.956710 8051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:11.958139 8036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:11.958288 8036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "13f2f770a5974db2a74b252cd24b58dd"
format_stamp: "Formatted at 2026-09-28 10:58:11 on dist-test-slave-djjw"
server_key: "cd1b31bcb294d4d6011b9001385023d1"
server_key_iv: "39ac56e94f34e4277a0862dd35fae463"
server_key_version: "encryptionkey@0"
I20260928 10:58:11.958482 8036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:11.994129 8036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:11.998239 8036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:11.998517 8036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:11.999469 8036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:12.000773 8036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:12.000847 8036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:12.000962 8036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:12.001013 8036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:12.029955 8036 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:44691
I20260928 10:58:12.030012 8163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:44691 every 8 connection(s)
I20260928 10:58:12.031243 8036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:12.033921 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8036
I20260928 10:58:12.034093 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:12.034528 4081 external_mini_cluster.cc:1428] Setting key e7311b9698befefc2b31ba2b127a09fb
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593091, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:12.050689 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:12.033924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:55697 (local address 127.3.252.126:41375)
0928 10:58:12.034119 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:12.034125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:12.035398 (+ 1273us) server_negotiation.cc:407] Connection header received
0928 10:58:12.037228 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:12.037233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:12.037315 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:12.037399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:12.038946 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.039674 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.041218 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.041612 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.041718 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:12.045004 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:12.045009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:12.045012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:12.045045 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:12.047671 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.048428 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.048433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.048435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.048562 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.048969 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.048972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.048974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.049255 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:12.049382 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:12.050140 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 10:58:12.050361 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:58:12.051733 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:12.034473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:41375 (local address 127.3.252.67:55697)
0928 10:58:12.035239 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:12.035283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:12.036973 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:12.037648 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:12.037656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:12.038209 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:12.038783 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:12.038801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.039821 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.039826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:12.041088 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:12.041099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.041853 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.041862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:12.042095 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:12.042655 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:12.042680 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:12.044788 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:12.047831 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:12.047840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:12.047855 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:12.048276 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:12.048665 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:12.048669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:12.048672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:12.048854 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:12.049382 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:12.049392 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:12.049736 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 10:58:12.050030 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 10:58:12.050583 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":327,"threads_started":1}
I20260928 10:58:12.053145 8164 heartbeater.cc:347] Connected to a master server at 127.3.252.126:41375
I20260928 10:58:12.053431 8164 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:12.054143 8164 heartbeater.cc:511] Master 127.3.252.126:41375 requested a full tablet report, sending...
I20260928 10:58:12.055704 7708 ts_manager.cc:194] Registered new tserver with Master: 13f2f770a5974db2a74b252cd24b58dd (127.3.252.67:44691)
I20260928 10:58:12.056490 7708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:55697
I20260928 10:58:12.061560 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[7663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 10:58:12.088809 7897 heartbeater.cc:503] Master 127.3.252.126:41375 was elected leader, sending a full tablet report...
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:12.112386 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:12.094512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.3.252.65:40835)
0928 10:58:12.098347 (+ 3835us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:12.098355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:12.098382 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:58:12.098440 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:12.098446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:12.098661 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:12.098799 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:12.100161 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.101167 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.102443 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.103013 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.103110 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:12.107028 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:12.107040 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:12.107055 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:12.107090 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:12.109760 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.110518 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.110525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.110527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.110646 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.110964 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.110967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.110969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.111302 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:12.111525 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:12.111820 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 10:58:12.112016 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3646,"thread_start_us":3497,"threads_started":1}
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:12.131489 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:12.116563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.3.252.126:41375)
0928 10:58:12.116771 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:12.116778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:12.116881 (+ 103us) server_negotiation.cc:407] Connection header received
0928 10:58:12.117021 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:12.117025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:12.117085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:12.117160 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:12.118440 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.119206 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.121121 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.121740 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.121882 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:12.126581 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:12.126586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:12.126590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:12.126627 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:12.129124 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.129802 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.129808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.129811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.129998 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.130314 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.130318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.130320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.130663 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:12.130852 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:12.131103 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 10:58:12.131265 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 10:58:12.132059 7708 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54730
I20260928 10:58:12.133083 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7769
I20260928 10:58:12.144886 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:12.134116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.3.252.65:40835)
0928 10:58:12.134287 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:12.134292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:12.134482 (+ 190us) server_negotiation.cc:407] Connection header received
0928 10:58:12.134630 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:12.134634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:12.134696 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:12.134798 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:12.136008 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.136682 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.137815 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.138301 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.138397 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:12.140476 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:12.140480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:12.140483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:12.140514 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:12.142790 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.143371 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.143374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.143377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.143482 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.143795 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.143798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.143800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.144034 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:12.144197 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:12.144474 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 10:58:12.144642 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:58:12.145547 7892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:12.227111 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7769
I20260928 10:58:12.240316 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7903
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:58:12.256191 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:12.241168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.3.252.66:40857)
0928 10:58:12.241711 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:12.241718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:12.241744 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:58:12.241811 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:12.241818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:12.242062 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:12.242217 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:12.243447 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.244446 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.245716 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.246321 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.246437 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:12.250648 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:12.250679 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:12.250695 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:12.250731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:12.253661 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.254321 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.254329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.254333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.254452 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.254808 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.254811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.254813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.255147 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:12.255368 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:12.255623 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 10:58:12.255816 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":268,"threads_started":1}
I20260928 10:58:12.257418 8026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:12.338074 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7903
I20260928 10:58:12.350932 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8036
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[7663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:58:12.366989 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:12.351966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.3.252.67:44691)
0928 10:58:12.352522 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:12.352529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:12.352560 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:58:12.352632 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:12.352641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:12.352868 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:12.353029 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:12.354274 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.355269 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.356477 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:12.357022 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:12.357131 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:12.361496 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:12.361510 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:12.361527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:12.361565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:12.364422 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.365088 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.365095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.365097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.365217 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:12.365501 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:12.365503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:12.365505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:12.365869 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:12.366119 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:12.366428 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 10:58:12.366629 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":282,"threads_started":1}
I20260928 10:58:12.368268 8159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:12.449657 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8036
I20260928 10:58:12.462664 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 7676
I20260928 10:58:12.463768 7738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:12.565488 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 7676
2026-09-28T10:58:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3670 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[8183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[8183](info): set up 2 sockets
Sep 28 10:58:12 dist-test-slave-djjw krb5kdc[8183](info): commencing operation
krb5kdc: starting...
W20260928 10:58:13.562336 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:13 dist-test-slave-djjw krb5kdc[8183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:13.759291 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:46503
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:46503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:14.069180 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:14.069484 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:14.069576 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:14.076902 8196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:14.077011 8196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:14.077055 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:14.077107 8196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:14.077153 8196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:14.087026 8196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:46503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:46503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:14.088248 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:14.089532 8196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:14.100631 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:14.100656 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:14.100597 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:14.101382 8196 server_base.cc:1034] running on GCE node
I20260928 10:58:14.102177 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:14.103507 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:14.104717 8196 hybrid_clock.cc:648] HybridClock initialized: now 1790593094104665 us; error 38 us; skew 500 ppm
I20260928 10:58:14.105041 8196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:14 dist-test-slave-djjw krb5kdc[8183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593094, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:14.113880 8196 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:14.115069 8196 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:14.115170 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:14.115507 8196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:14.118489 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "5d15d475848943cfb1a5cba0b24b0aa4"
format_stamp: "Formatted at 2026-09-28 10:58:14 on dist-test-slave-djjw"
server_key: "3cd87274747bd85035e0420df03eee41"
server_key_iv: "b642e4e276fc9485763c37ac71b7d094"
server_key_version: "encryptionkey@0"
I20260928 10:58:14.119361 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "5d15d475848943cfb1a5cba0b24b0aa4"
format_stamp: "Formatted at 2026-09-28 10:58:14 on dist-test-slave-djjw"
server_key: "3cd87274747bd85035e0420df03eee41"
server_key_iv: "b642e4e276fc9485763c37ac71b7d094"
server_key_version: "encryptionkey@0"
I20260928 10:58:14.126267 8196 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:58:14.130492 8210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:14.131831 8196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:58:14.131978 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "5d15d475848943cfb1a5cba0b24b0aa4"
format_stamp: "Formatted at 2026-09-28 10:58:14 on dist-test-slave-djjw"
server_key: "3cd87274747bd85035e0420df03eee41"
server_key_iv: "b642e4e276fc9485763c37ac71b7d094"
server_key_version: "encryptionkey@0"
I20260928 10:58:14.132186 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:14.151844 8196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:14.155970 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:14.156193 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:14.173626 8196 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:46503
I20260928 10:58:14.173645 8261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:46503 every 8 connection(s)
I20260928 10:58:14.174540 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:14.176819 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8196
I20260928 10:58:14.177045 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:14.177793 4081 external_mini_cluster.cc:1428] Setting key 16f2585e5e51f27a1fca6827da14c46b
I20260928 10:58:14.178701 8262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:58:14 dist-test-slave-djjw krb5kdc[8183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:14.192742 8262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4: Bootstrap starting.
I20260928 10:58:14.195952 8262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:14.197167 8262 log.cc:824] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:14.199913 8262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4: No bootstrap required, opened a new log
I20260928 10:58:14.199928 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:14.179849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.3.252.126:46503)
0928 10:58:14.180584 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:14.180596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:14.180635 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:58:14.182149 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:14.182172 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:14.182603 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:14.182944 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:14.184721 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:14.186086 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:14.187740 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:14.188381 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:14.188515 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:14.193077 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:14.193086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:14.193102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:14.193142 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:14.195905 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:14.196658 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:14.196664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:14.196670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:14.196828 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:14.197214 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:14.197218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:14.197219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:14.197799 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:14.198146 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:14.198586 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 10:58:14.198974 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":333,"threads_started":1}
I20260928 10:58:14.207315 8262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46503 } }
I20260928 10:58:14.207683 8262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:14.207777 8262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d15d475848943cfb1a5cba0b24b0aa4, State: Initialized, Role: FOLLOWER
I20260928 10:58:14.208352 8262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46503 } }
I20260928 10:58:14.208588 8262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:14.208726 8262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:14.208918 8262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:14.211081 8262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46503 } }
I20260928 10:58:14.211588 8262 leader_election.cc:302] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d15d475848943cfb1a5cba0b24b0aa4; no voters:
I20260928 10:58:14.212167 8262 leader_election.cc:288] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:14.212306 8267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:14.212980 8267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d15d475848943cfb1a5cba0b24b0aa4, State: Running, Role: LEADER
I20260928 10:58:14.213511 8267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46503 } }
I20260928 10:58:14.214311 8262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:14.216122 8269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d15d475848943cfb1a5cba0b24b0aa4. Latest consensus state: current_term: 1 leader_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46503 } } }
I20260928 10:58:14.216464 8269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:14.217111 8268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d15d475848943cfb1a5cba0b24b0aa4" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46503 } } }
I20260928 10:58:14.217335 8273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:14.217396 8268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:14.223539 8273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:14.232200 8273 catalog_manager.cc:1347] Generated new cluster ID: c2d51f3ed77e402ca8c8dbf3c1729fe6
I20260928 10:58:14.232317 8273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:14.263664 8273 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:14.264978 8273 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:14.273295 8273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4: Generated new TSK 0
I20260928 10:58:14.274153 8273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:58:14.325518 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:46503
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:14.649263 8288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:14.649535 8288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:14.649657 8288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:14.656894 8288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:14.657002 8288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:14.657152 8288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:14.667184 8288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:14.668581 8288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:14.670009 8288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:14.682438 8294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:14.682467 8293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:14.682500 8296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:14.683131 8288 server_base.cc:1034] running on GCE node
I20260928 10:58:14.683785 8288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:14.684739 8288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:14.685925 8288 hybrid_clock.cc:648] HybridClock initialized: now 1790593094685873 us; error 39 us; skew 500 ppm
I20260928 10:58:14.686234 8288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:14 dist-test-slave-djjw krb5kdc[8183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593094, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:14.689260 8288 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:58:14.690465 8288 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:14.690567 8288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:14.690871 8288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:14.693769 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "6ddf57bcd8734e449b2383ae70459172"
format_stamp: "Formatted at 2026-09-28 10:58:14 on dist-test-slave-djjw"
server_key: "4812100cee7ce6ed6560ccc315df538c"
server_key_iv: "0e0dfb131ac6ef28a49cd8889d60e1fd"
server_key_version: "encryptionkey@0"
I20260928 10:58:14.694684 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "6ddf57bcd8734e449b2383ae70459172"
format_stamp: "Formatted at 2026-09-28 10:58:14 on dist-test-slave-djjw"
server_key: "4812100cee7ce6ed6560ccc315df538c"
server_key_iv: "0e0dfb131ac6ef28a49cd8889d60e1fd"
server_key_version: "encryptionkey@0"
I20260928 10:58:14.700109 8288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:14.703897 8302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:14.705361 8288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:14.705523 8288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "6ddf57bcd8734e449b2383ae70459172"
format_stamp: "Formatted at 2026-09-28 10:58:14 on dist-test-slave-djjw"
server_key: "4812100cee7ce6ed6560ccc315df538c"
server_key_iv: "0e0dfb131ac6ef28a49cd8889d60e1fd"
server_key_version: "encryptionkey@0"
I20260928 10:58:14.705739 8288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:14.725278 8288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:14.729382 8288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:14.729646 8288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:14.730636 8288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:14.731946 8288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:14.732036 8288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:14.732149 8288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:14.732215 8288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:14.761482 8288 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:41623
I20260928 10:58:14.761516 8414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:41623 every 8 connection(s)
I20260928 10:58:14.762337 8288 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:14.766000 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8288
I20260928 10:58:14.766215 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:14.766987 4081 external_mini_cluster.cc:1428] Setting key 62383a26c456ccc74f4ae6e93ff579a6
Sep 28 10:58:14 dist-test-slave-djjw krb5kdc[8183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593094, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:14.782976 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:14.764634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:55673 (local address 127.3.252.126:46503)
0928 10:58:14.765074 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:14.765080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:14.766192 (+ 1112us) server_negotiation.cc:407] Connection header received
0928 10:58:14.768314 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:14.768319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:14.768420 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:14.768534 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:14.770438 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:14.771130 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:14.773146 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:14.773598 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:14.773688 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:14.777155 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:14.777161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:14.777166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:14.777204 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:14.779992 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:14.780705 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:14.780710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:14.780714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:14.780843 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:14.781193 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:14.781197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:14.781200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:14.781536 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:14.781685 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:14.782372 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 10:58:14.782599 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
I20260928 10:58:14.783841 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:14.765193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:46503 (local address 127.3.252.65:55673)
0928 10:58:14.765991 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:14.766039 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:14.768070 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:14.768717 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:14.768725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:14.769409 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:14.770253 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:14.770278 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:14.771275 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:14.771280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:14.773016 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:14.773030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:14.773798 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:14.773804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:14.774091 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:14.774764 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:14.774786 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:14.776922 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:14.780121 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:14.780128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:14.780142 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:14.780580 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:14.780931 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:14.780934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:14.780936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:14.781095 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:14.781645 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:14.781651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:14.782014 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 10:58:14.782245 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 10:58:14.782713 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":276,"threads_started":1}
I20260928 10:58:14.785320 8415 heartbeater.cc:347] Connected to a master server at 127.3.252.126:46503
I20260928 10:58:14.785665 8415 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 10:58:14.786540 8415 heartbeater.cc:511] Master 127.3.252.126:46503 requested a full tablet report, sending...
I20260928 10:58:14.788581 8227 ts_manager.cc:194] Registered new tserver with Master: 6ddf57bcd8734e449b2383ae70459172 (127.3.252.65:41623)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
I20260928 10:58:14.790122 8227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:55673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:58:14.811527 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:46503
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:15.127753 8421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:15.128041 8421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:15.128166 8421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:15.135350 8421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:15.135461 8421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:15.135633 8421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:15.145747 8421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:15.147085 8421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:15.148383 8421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:15.161024 8426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:15.161024 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:15.161082 8429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:15.161733 8421 server_base.cc:1034] running on GCE node
I20260928 10:58:15.162391 8421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:15.163357 8421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:15.164574 8421 hybrid_clock.cc:648] HybridClock initialized: now 1790593095164531 us; error 32 us; skew 500 ppm
I20260928 10:58:15.164912 8421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:15 dist-test-slave-djjw krb5kdc[8183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593095, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:15.167970 8421 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:58:15.169158 8421 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:15.169260 8421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:15.169618 8421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:15.172691 8421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "ac86cc8670724aecba6b2b55caacff76"
format_stamp: "Formatted at 2026-09-28 10:58:15 on dist-test-slave-djjw"
server_key: "cc8fda3bbdf72be668f7e71afc557058"
server_key_iv: "ffba5b5108247abd84483cb6c6f7791f"
server_key_version: "encryptionkey@0"
I20260928 10:58:15.173492 8421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "ac86cc8670724aecba6b2b55caacff76"
format_stamp: "Formatted at 2026-09-28 10:58:15 on dist-test-slave-djjw"
server_key: "cc8fda3bbdf72be668f7e71afc557058"
server_key_iv: "ffba5b5108247abd84483cb6c6f7791f"
server_key_version: "encryptionkey@0"
I20260928 10:58:15.178771 8421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:58:15.182402 8435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:15.183930 8421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 10:58:15.184095 8421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "ac86cc8670724aecba6b2b55caacff76"
format_stamp: "Formatted at 2026-09-28 10:58:15 on dist-test-slave-djjw"
server_key: "cc8fda3bbdf72be668f7e71afc557058"
server_key_iv: "ffba5b5108247abd84483cb6c6f7791f"
server_key_version: "encryptionkey@0"
I20260928 10:58:15.184321 8421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:15.205013 8421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:15.208935 8421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:15.209190 8421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:15.210173 8421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:15.211544 8421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:15.211629 8421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:15.211731 8421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:15.211790 8421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:15.241349 8421 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:44431
I20260928 10:58:15.241398 8547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:44431 every 8 connection(s)
I20260928 10:58:15.242224 8421 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:15.249818 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8421
I20260928 10:58:15.250029 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:15.250483 4081 external_mini_cluster.cc:1428] Setting key e6a5f01197dd01cc42ddcd30d67f5a72
Sep 28 10:58:15 dist-test-slave-djjw krb5kdc[8183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593095, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:15.261694 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.244568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:46863 (local address 127.3.252.126:46503)
0928 10:58:15.244735 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.244739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.245844 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 10:58:15.247584 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.247590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.247669 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.247757 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:15.249254 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.250062 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.251782 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.252321 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.252441 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.256032 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:15.256038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:15.256041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:15.256076 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:15.258646 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:15.259401 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:15.259406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:15.259408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:15.259544 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:15.259967 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:15.259971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:15.259973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:15.260246 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:15.260383 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.261127 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.261353 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:58:15.262760 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.245013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:46503 (local address 127.3.252.66:46863)
0928 10:58:15.245682 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:15.245718 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:15.247317 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:15.247985 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:15.247993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:15.248507 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:15.249082 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.249101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.250212 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.250216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.251644 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.251657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.252559 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.252567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.252801 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.253487 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:15.253512 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:15.255779 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:15.258799 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:15.258809 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:15.258828 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:15.259277 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:15.259655 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:15.259660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:15.259664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:15.259856 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:15.260383 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:15.260392 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:15.260758 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 10:58:15.261024 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 10:58:15.261598 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":242,"threads_started":1}
I20260928 10:58:15.264271 8548 heartbeater.cc:347] Connected to a master server at 127.3.252.126:46503
I20260928 10:58:15.264644 8548 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:15.265475 8548 heartbeater.cc:511] Master 127.3.252.126:46503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 10:58:15.267179 8227 ts_manager.cc:194] Registered new tserver with Master: ac86cc8670724aecba6b2b55caacff76 (127.3.252.66:44431)
I20260928 10:58:15.268034 8227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:46863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:58:15.293268 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:46503
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:15.623485 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:15.623895 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:15.624050 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:15.633394 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:15.633533 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:15.633713 8553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:15.645371 8553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:15.646787 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:15.648096 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:15.660948 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:15.660912 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:15.661265 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:15.663182 8553 server_base.cc:1034] running on GCE node
I20260928 10:58:15.663836 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:15.664829 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:15.666020 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790593095665978 us; error 32 us; skew 500 ppm
I20260928 10:58:15.666329 8553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:15 dist-test-slave-djjw krb5kdc[8183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593095, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:15.669394 8553 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:58:15.670651 8553 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:15.670745 8553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:15.671051 8553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:15.674015 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "59a61183bb64443697f60b7f5791fa25"
format_stamp: "Formatted at 2026-09-28 10:58:15 on dist-test-slave-djjw"
server_key: "0ff9ab2f316ace27f2c6dc9152274ecc"
server_key_iv: "a0ba3765dcacec1d777c3b9d2e7ec258"
server_key_version: "encryptionkey@0"
I20260928 10:58:15.674800 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "59a61183bb64443697f60b7f5791fa25"
format_stamp: "Formatted at 2026-09-28 10:58:15 on dist-test-slave-djjw"
server_key: "0ff9ab2f316ace27f2c6dc9152274ecc"
server_key_iv: "a0ba3765dcacec1d777c3b9d2e7ec258"
server_key_version: "encryptionkey@0"
I20260928 10:58:15.679924 8553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:15.683483 8567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:15.684890 8553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 10:58:15.685065 8553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "59a61183bb64443697f60b7f5791fa25"
format_stamp: "Formatted at 2026-09-28 10:58:15 on dist-test-slave-djjw"
server_key: "0ff9ab2f316ace27f2c6dc9152274ecc"
server_key_iv: "a0ba3765dcacec1d777c3b9d2e7ec258"
server_key_version: "encryptionkey@0"
I20260928 10:58:15.685314 8553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:15.706904 8553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:15.711269 8553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:15.711519 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:15.712474 8553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:15.713989 8553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:15.714083 8553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:15.714190 8553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:15.714273 8553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:15.742807 8553 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:36589
I20260928 10:58:15.742844 8679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:36589 every 8 connection(s)
I20260928 10:58:15.743660 8553 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:15.752475 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8553
I20260928 10:58:15.752658 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:15.753074 4081 external_mini_cluster.cc:1428] Setting key 25d381051b40e40dd8ecf6bb780d64e6
Sep 28 10:58:15 dist-test-slave-djjw krb5kdc[8183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593095, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:15.761679 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.745911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:36927 (local address 127.3.252.126:46503)
0928 10:58:15.746071 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.746075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.747200 (+ 1125us) server_negotiation.cc:407] Connection header received
0928 10:58:15.748875 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.748879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.748944 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.749029 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:15.750421 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.751126 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.752807 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.753226 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.753330 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.756757 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:15.756762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:15.756765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:15.756797 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:15.759072 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:15.759714 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:15.759717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:15.759720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:15.759842 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:15.760156 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:15.760159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:15.760161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:15.760405 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:15.760509 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.761152 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.761357 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:58:15.762681 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.746405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:46503 (local address 127.3.252.67:36927)
0928 10:58:15.747047 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:15.747083 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:15.748631 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:15.749196 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:15.749203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:15.749671 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:15.750270 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.750287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.751240 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.751244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.752676 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.752691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.753441 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.753449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.753671 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.754360 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:15.754389 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:15.756555 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:15.759188 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:15.759194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:15.759208 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:15.759623 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:15.759920 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:15.759922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:15.759925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:15.760077 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:15.760502 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:15.760507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:15.760790 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 10:58:15.761016 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 10:58:15.761548 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":261,"threads_started":1}
I20260928 10:58:15.764099 8680 heartbeater.cc:347] Connected to a master server at 127.3.252.126:46503
I20260928 10:58:15.764424 8680 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:15.765062 8680 heartbeater.cc:511] Master 127.3.252.126:46503 requested a full tablet report, sending...
I20260928 10:58:15.766625 8227 ts_manager.cc:194] Registered new tserver with Master: 59a61183bb64443697f60b7f5791fa25 (127.3.252.67:36589)
I20260928 10:58:15.767549 8227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:36927
I20260928 10:58:15.768187 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:58:15.784567 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.772446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.3.252.126:46503)
0928 10:58:15.772585 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.772590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.773053 (+ 463us) server_negotiation.cc:407] Connection header received
0928 10:58:15.773234 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.773238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.773299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.773378 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:15.774778 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.775467 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.776722 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.777151 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.777257 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.779459 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:15.779464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:15.779467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:15.779497 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:15.782144 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:15.782772 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:15.782778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:15.782782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:15.782911 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:15.783245 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:15.783250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:15.783253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:15.783554 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:15.783770 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.784091 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.784287 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 10:58:15.793007 8415 heartbeater.cc:503] Master 127.3.252.126:46503 was elected leader, sending a full tablet report...
I20260928 10:58:15.788504 8227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:58:15.795433 8227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:15.825583 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.807111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.3.252.66:44431)
0928 10:58:15.816002 (+ 8891us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.816013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.816054 (+ 41us) server_negotiation.cc:407] Connection header received
0928 10:58:15.816176 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.816185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.816482 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.816793 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:15.817786 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.819609 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.822516 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.823904 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.824045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.824257 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.825003 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.825171 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8751,"thread_start_us":8656,"threads_started":1}
I20260928 10:58:15.826331 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.806974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:44431 (local address 127.0.0.1:39658)
0928 10:58:15.814223 (+ 7249us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:15.814278 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:15.814457 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:15.817082 (+ 2625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:15.817091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:15.817134 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:15.817560 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.817573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.819785 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.819791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.822329 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.822341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.824331 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.824340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.824546 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.824655 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 10:58:15.824804 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 10:58:15.825896 (+ 1092us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6969,"mutex_wait_us":880,"thread_start_us":6807,"threads_started":1}
I20260928 10:58:15.828611 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.808080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:41623 (local address 127.0.0.1:32802)
0928 10:58:15.820746 (+ 12666us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:15.820766 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:15.820871 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:15.821779 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:15.821784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:15.821814 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:15.822247 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.822267 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.824135 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.824140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.826082 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.826094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.827665 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.827670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.827829 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.827860 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:58:15.827943 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 10:58:15.828059 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12561,"mutex_wait_us":79,"thread_start_us":12418,"threads_started":1}
I20260928 10:58:15.828491 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.808702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.3.252.65:41623)
0928 10:58:15.820306 (+ 11604us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.820316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.820996 (+ 680us) server_negotiation.cc:407] Connection header received
0928 10:58:15.821094 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.821102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.821369 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.821545 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:15.822433 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.823983 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.826220 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.827471 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.827604 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.827808 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.828017 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.828169 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11425,"thread_start_us":11281,"threads_started":1}
I20260928 10:58:15.829476 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.807135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:36589 (local address 127.0.0.1:52816)
0928 10:58:15.819493 (+ 12358us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:15.819513 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:15.819620 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:15.822710 (+ 3090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:15.822714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:15.822745 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:15.823045 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.823053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.824487 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.824492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.826667 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.826680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.828599 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.828605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.828890 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.828935 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 10:58:15.829011 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 10:58:15.829127 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12223,"mutex_wait_us":55,"thread_start_us":12555,"threads_started":1}
I20260928 10:58:15.830199 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.807573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.3.252.67:36589)
0928 10:58:15.822209 (+ 14636us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.822217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.822261 (+ 44us) server_negotiation.cc:407] Connection header received
0928 10:58:15.822353 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.822360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.822545 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.822702 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:15.823231 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.824327 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.826833 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.828139 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.828250 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.828451 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.829687 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.829870 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14466,"thread_start_us":14332,"threads_started":1}
I20260928 10:58:15.830391 8483 tablet_service.cc:1467] Processing CreateTablet for tablet a44f4b265df64904a2738f4f3e02ff37 (DEFAULT_TABLE table=test-table [id=585f3c3df816486bbd8d0ccbfc8cd6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:15.831869 8483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44f4b265df64904a2738f4f3e02ff37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:15.832820 8350 tablet_service.cc:1467] Processing CreateTablet for tablet a44f4b265df64904a2738f4f3e02ff37 (DEFAULT_TABLE table=test-table [id=585f3c3df816486bbd8d0ccbfc8cd6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:15.834203 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44f4b265df64904a2738f4f3e02ff37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:15.836369 8615 tablet_service.cc:1467] Processing CreateTablet for tablet a44f4b265df64904a2738f4f3e02ff37 (DEFAULT_TABLE table=test-table [id=585f3c3df816486bbd8d0ccbfc8cd6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:15.837450 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44f4b265df64904a2738f4f3e02ff37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:15.846095 8698 tablet_bootstrap.cc:492] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Bootstrap starting.
I20260928 10:58:15.847817 8699 tablet_bootstrap.cc:492] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Bootstrap starting.
I20260928 10:58:15.849125 8700 tablet_bootstrap.cc:492] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Bootstrap starting.
I20260928 10:58:15.850157 8698 tablet_bootstrap.cc:654] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:15.851040 8699 tablet_bootstrap.cc:654] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:15.851094 8698 log.cc:824] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:15.852185 8699 log.cc:824] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:15.852257 8700 tablet_bootstrap.cc:654] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:15.853233 8700 log.cc:824] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:15.853744 8698 tablet_bootstrap.cc:492] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: No bootstrap required, opened a new log
I20260928 10:58:15.854076 8698 ts_tablet_manager.cc:1397] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 10:58:15.855050 8699 tablet_bootstrap.cc:492] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: No bootstrap required, opened a new log
I20260928 10:58:15.855376 8699 ts_tablet_manager.cc:1397] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 10:58:15.856123 8700 tablet_bootstrap.cc:492] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: No bootstrap required, opened a new log
I20260928 10:58:15.856395 8700 ts_tablet_manager.cc:1397] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20260928 10:58:15.861261 8698 raft_consensus.cc:348] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.861610 8698 raft_consensus.cc:374] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:15.861742 8698 raft_consensus.cc:729] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86cc8670724aecba6b2b55caacff76, State: Initialized, Role: FOLLOWER
I20260928 10:58:15.862309 8698 consensus_queue.cc:260] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.863394 8548 heartbeater.cc:503] Master 127.3.252.126:46503 was elected leader, sending a full tablet report...
I20260928 10:58:15.863873 8698 ts_tablet_manager.cc:1428] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260928 10:58:15.865424 8699 raft_consensus.cc:348] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.865863 8699 raft_consensus.cc:374] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:15.865957 8699 raft_consensus.cc:729] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ddf57bcd8734e449b2383ae70459172, State: Initialized, Role: FOLLOWER
I20260928 10:58:15.866609 8699 consensus_queue.cc:260] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.867282 8700 raft_consensus.cc:348] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.867756 8700 raft_consensus.cc:374] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:15.867889 8700 raft_consensus.cc:729] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a61183bb64443697f60b7f5791fa25, State: Initialized, Role: FOLLOWER
I20260928 10:58:15.868597 8699 ts_tablet_manager.cc:1428] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 10:58:15.868498 8700 consensus_queue.cc:260] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.869653 8680 heartbeater.cc:503] Master 127.3.252.126:46503 was elected leader, sending a full tablet report...
I20260928 10:58:15.870488 8700 ts_tablet_manager.cc:1428] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 10:58:15.957386 8704 raft_consensus.cc:482] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:15.957686 8704 raft_consensus.cc:504] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.959414 8704 leader_election.cc:288] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59a61183bb64443697f60b7f5791fa25 (127.3.252.67:36589), 6ddf57bcd8734e449b2383ae70459172 (127.3.252.65:41623)
I20260928 10:58:15.966104 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.959968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:60779 (local address 127.3.252.67:36589)
0928 10:58:15.960174 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.960181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.960867 (+ 686us) server_negotiation.cc:407] Connection header received
0928 10:58:15.961062 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.961066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.961133 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.961208 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:15.961881 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.962580 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.964429 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.965194 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.965348 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.965432 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.965708 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.965867 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:58:15.966109 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.959852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:59363 (local address 127.3.252.65:41623)
0928 10:58:15.960078 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:15.960085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:15.960900 (+ 815us) server_negotiation.cc:407] Connection header received
0928 10:58:15.961070 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:15.961074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:15.961130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:15.961196 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:15.961754 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.962517 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.964440 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.965165 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.965298 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.965377 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:15.965711 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 10:58:15.965857 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 10:58:15.966317 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.959839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:41623 (local address 127.3.252.66:59363)
0928 10:58:15.960782 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:15.960800 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:15.960940 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:15.961230 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:15.961234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:15.961268 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:15.961607 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.961617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.962639 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.962644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.964281 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.964293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.965387 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.965394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.965563 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.965589 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:58:15.965669 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 10:58:15.965852 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":53,"spinlock_wait_cycles":1280,"thread_start_us":402,"threads_started":1}
I20260928 10:58:15.966421 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:15.959706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:36589 (local address 127.3.252.66:60779)
0928 10:58:15.960745 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:15.960761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:15.960899 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:15.961236 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:15.961240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:15.961278 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:15.961705 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.961717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.962711 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.962715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.964271 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:15.964284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:15.965418 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:15.965425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:15.965567 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:15.965594 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 10:58:15.965670 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 10:58:15.965856 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":75,"spinlock_wait_cycles":1280,"thread_start_us":764,"threads_started":1}
I20260928 10:58:15.967311 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44f4b265df64904a2738f4f3e02ff37" candidate_uuid: "ac86cc8670724aecba6b2b55caacff76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ddf57bcd8734e449b2383ae70459172" is_pre_election: true
I20260928 10:58:15.967545 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44f4b265df64904a2738f4f3e02ff37" candidate_uuid: "ac86cc8670724aecba6b2b55caacff76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a61183bb64443697f60b7f5791fa25" is_pre_election: true
I20260928 10:58:15.967936 8370 raft_consensus.cc:2454] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86cc8670724aecba6b2b55caacff76 in term 0.
I20260928 10:58:15.967988 8635 raft_consensus.cc:2454] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86cc8670724aecba6b2b55caacff76 in term 0.
I20260928 10:58:15.968585 8437 leader_election.cc:302] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ddf57bcd8734e449b2383ae70459172, ac86cc8670724aecba6b2b55caacff76; no voters:
I20260928 10:58:15.968953 8704 raft_consensus.cc:2781] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:15.969103 8704 raft_consensus.cc:482] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:15.969177 8704 raft_consensus.cc:3037] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:15.971536 8704 raft_consensus.cc:504] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.972136 8704 leader_election.cc:288] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [CANDIDATE]: Term 1 election: Requested vote from peers 59a61183bb64443697f60b7f5791fa25 (127.3.252.67:36589), 6ddf57bcd8734e449b2383ae70459172 (127.3.252.65:41623)
I20260928 10:58:15.972558 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44f4b265df64904a2738f4f3e02ff37" candidate_uuid: "ac86cc8670724aecba6b2b55caacff76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a61183bb64443697f60b7f5791fa25"
I20260928 10:58:15.972564 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44f4b265df64904a2738f4f3e02ff37" candidate_uuid: "ac86cc8670724aecba6b2b55caacff76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ddf57bcd8734e449b2383ae70459172"
I20260928 10:58:15.972735 8370 raft_consensus.cc:3037] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:15.972728 8635 raft_consensus.cc:3037] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:15.974848 8635 raft_consensus.cc:2454] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86cc8670724aecba6b2b55caacff76 in term 1.
I20260928 10:58:15.974855 8370 raft_consensus.cc:2454] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86cc8670724aecba6b2b55caacff76 in term 1.
I20260928 10:58:15.975203 8437 leader_election.cc:302] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ddf57bcd8734e449b2383ae70459172, ac86cc8670724aecba6b2b55caacff76; no voters:
I20260928 10:58:15.975485 8704 raft_consensus.cc:2781] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:15.975883 8704 raft_consensus.cc:686] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 1 LEADER]: Becoming Leader. State: Replica: ac86cc8670724aecba6b2b55caacff76, State: Running, Role: LEADER
I20260928 10:58:15.976428 8704 consensus_queue.cc:237] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } }
I20260928 10:58:15.980785 8226 catalog_manager.cc:5495] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac86cc8670724aecba6b2b55caacff76 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "ac86cc8670724aecba6b2b55caacff76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac86cc8670724aecba6b2b55caacff76" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 } health_report { overall_health: UNKNOWN } } }
W20260928 10:58:15.993990 8549 tablet.cc:2369] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:58:15.994890 8681 tablet.cc:2369] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:58:16.014479 8416 tablet.cc:2369] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:58:16.018813 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:16.013658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.3.252.66:44431)
0928 10:58:16.013878 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:16.013884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:16.013949 (+ 65us) server_negotiation.cc:407] Connection header received
0928 10:58:16.014192 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:16.014197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:16.014296 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:16.014482 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:16.014822 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.015674 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.016927 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.017299 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.017396 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:16.017790 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:16.018337 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:16.018441 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:16.018494 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 10:58:16.018557 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:58:16.026510 8635 raft_consensus.cc:1261] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 1 FOLLOWER]: Refusing update from remote peer ac86cc8670724aecba6b2b55caacff76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:16.026503 8370 raft_consensus.cc:1261] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 1 FOLLOWER]: Refusing update from remote peer ac86cc8670724aecba6b2b55caacff76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:16.027387 8709 consensus_queue.cc:1035] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ddf57bcd8734e449b2383ae70459172" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:16.027695 8704 consensus_queue.cc:1035] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59a61183bb64443697f60b7f5791fa25" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 36589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:16.040781 8712 mvcc.cc:205] Tried to move back new op lower bound from 7334269321315524608 to 7334269321123381248. Current Snapshot: MvccSnapshot[applied={T|T < 7334269321315524608}]
I20260928 10:58:16.058866 8226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:16.059155 8226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:16.062704 8226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d15d475848943cfb1a5cba0b24b0aa4: Sending DeleteTablet for 3 replicas of tablet a44f4b265df64904a2738f4f3e02ff37
I20260928 10:58:16.063880 8483 tablet_service.cc:1514] Processing DeleteTablet for tablet a44f4b265df64904a2738f4f3e02ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:16 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:39658
I20260928 10:58:16.063949 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet a44f4b265df64904a2738f4f3e02ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:16 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:52816
I20260928 10:58:16.064024 8350 tablet_service.cc:1514] Processing DeleteTablet for tablet a44f4b265df64904a2738f4f3e02ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:16 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:32802
I20260928 10:58:16.067652 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8288
I20260928 10:58:16.076447 8723 tablet_replica.cc:330] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: stopping tablet replica
I20260928 10:58:16.076941 8723 raft_consensus.cc:2229] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:16.077255 8724 tablet_replica.cc:330] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: stopping tablet replica
I20260928 10:58:16.077458 8723 raft_consensus.cc:2258] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:16.077667 8724 raft_consensus.cc:2229] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:16.078141 8724 raft_consensus.cc:2258] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:16.078881 8722 tablet_replica.cc:330] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: stopping tablet replica
I20260928 10:58:16.079293 8722 raft_consensus.cc:2229] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:16.079931 8722 raft_consensus.cc:2258] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:16.080386 8723 ts_tablet_manager.cc:1905] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:16.081791 8722 ts_tablet_manager.cc:1905] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:16.084908 8724 ts_tablet_manager.cc:1905] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:16.087757 8723 ts_tablet_manager.cc:1918] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 10:58:16 dist-test-slave-djjw krb5kdc[8183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:16.087997 8723 log.cc:1196] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/a44f4b265df64904a2738f4f3e02ff37
I20260928 10:58:16.088510 8723 ts_tablet_manager.cc:1939] T a44f4b265df64904a2738f4f3e02ff37 P 59a61183bb64443697f60b7f5791fa25: Deleting consensus metadata
I20260928 10:58:16.089459 8722 ts_tablet_manager.cc:1918] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:16.089645 8722 log.cc:1196] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/a44f4b265df64904a2738f4f3e02ff37
I20260928 10:58:16.090286 8722 ts_tablet_manager.cc:1939] T a44f4b265df64904a2738f4f3e02ff37 P ac86cc8670724aecba6b2b55caacff76: Deleting consensus metadata
I20260928 10:58:16.090399 8214 catalog_manager.cc:4841] TS 59a61183bb64443697f60b7f5791fa25 (127.3.252.67:36589): tablet a44f4b265df64904a2738f4f3e02ff37 (table test-table [id=585f3c3df816486bbd8d0ccbfc8cd6b1]) successfully deleted
I20260928 10:58:16.091872 8213 catalog_manager.cc:4841] TS ac86cc8670724aecba6b2b55caacff76 (127.3.252.66:44431): tablet a44f4b265df64904a2738f4f3e02ff37 (table test-table [id=585f3c3df816486bbd8d0ccbfc8cd6b1]) successfully deleted
I20260928 10:58:16.094813 8724 ts_tablet_manager.cc:1918] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:16.094946 8724 log.cc:1196] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/a44f4b265df64904a2738f4f3e02ff37
I20260928 10:58:16.095415 8724 ts_tablet_manager.cc:1939] T a44f4b265df64904a2738f4f3e02ff37 P 6ddf57bcd8734e449b2383ae70459172: Deleting consensus metadata
I20260928 10:58:16.096869 8212 catalog_manager.cc:4841] TS 6ddf57bcd8734e449b2383ae70459172 (127.3.252.65:41623): tablet a44f4b265df64904a2738f4f3e02ff37 (table test-table [id=585f3c3df816486bbd8d0ccbfc8cd6b1]) successfully deleted
I20260928 10:58:16.101867 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:16.068904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.3.252.65:41623)
0928 10:58:16.069085 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:16.069092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:16.076648 (+ 7556us) server_negotiation.cc:407] Connection header received
0928 10:58:16.076834 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:16.076839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:16.076928 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:16.077032 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:16.079603 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.080660 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.082431 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.082979 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.083069 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:16.095131 (+ 12062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:16.095147 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:16.095166 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:16.095211 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:16.098943 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:16.099733 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:16.099742 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:16.099746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:16.099923 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:16.100261 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:16.100266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:16.100269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:16.100730 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:16.101046 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:16.101246 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 10:58:16.101463 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:58:16.102592 8410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:16.196267 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8288
I20260928 10:58:16.210824 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8421
Sep 28 10:58:16 dist-test-slave-djjw krb5kdc[8183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:58:16.226483 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:16.211708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.3.252.66:44431)
0928 10:58:16.211906 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:16.211911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:16.211931 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:58:16.212100 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:16.212106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:16.212173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:16.212258 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:16.213528 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.214470 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.215830 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.216256 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.216383 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:16.220487 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:16.220502 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:16.220519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:16.220558 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:16.223627 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:16.224449 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:16.224456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:16.224459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:16.224609 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:16.224993 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:16.224996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:16.224999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:16.225383 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:16.225671 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:16.225994 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 10:58:16.226159 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:58:16.227180 8543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:16.338289 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8421
I20260928 10:58:16.352906 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8553
Sep 28 10:58:16 dist-test-slave-djjw krb5kdc[8183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:58:16.367269 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:16.353654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.3.252.67:36589)
0928 10:58:16.353820 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:16.353850 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:16.353872 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:58:16.354013 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:16.354017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:16.354077 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:16.354163 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:16.355405 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.356127 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.357297 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:16.357693 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:16.357782 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:16.361892 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:16.361907 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:16.361924 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:16.361959 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:16.364543 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:16.365172 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:16.365179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:16.365181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:16.365334 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:16.365680 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:16.365684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:16.365686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:16.366214 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:16.366489 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:16.366762 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 10:58:16.366954 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:58:16.367908 8675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:16.455009 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8553
I20260928 10:58:16.468112 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8196
I20260928 10:58:16.469136 8257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:16.574324 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8196
2026-09-28T10:58:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4008 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T10:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:16Z Disabled control of system clock
I20260928 10:58:16.633124 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41073
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:41073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 10:58:16.941963 8733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:16.942248 8733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:16.942369 8733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:16.949534 8733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:16.949635 8733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:16.949678 8733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:16.949751 8733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:16.949801 8733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:16.959110 8733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:41073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:16.960430 8733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:16.962244 8733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:16.973793 8738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:16.977801 8741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:16.977928 8739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:16.977950 8733 server_base.cc:1034] running on GCE node
I20260928 10:58:16.979142 8733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:16.980540 8733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:16.981753 8733 hybrid_clock.cc:648] HybridClock initialized: now 1790593096981697 us; error 40 us; skew 500 ppm
I20260928 10:58:16.982124 8733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:16.984365 8733 webserver.cc:460] Webserver started at http://127.3.252.126:45791/ using document root <none> and password file <none>
I20260928 10:58:16.985041 8733 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:16.985129 8733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:16.985440 8733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:16.988293 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "2514964d10644f9789d756c377d69f5f"
format_stamp: "Formatted at 2026-09-28 10:58:16 on dist-test-slave-djjw"
server_key: "40149abe2066aeb9f1895a94c66ced29"
server_key_iv: "9fe263328dd61adaf5dbde76fea1a136"
server_key_version: "encryptionkey@0"
I20260928 10:58:16.989059 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "2514964d10644f9789d756c377d69f5f"
format_stamp: "Formatted at 2026-09-28 10:58:16 on dist-test-slave-djjw"
server_key: "40149abe2066aeb9f1895a94c66ced29"
server_key_iv: "9fe263328dd61adaf5dbde76fea1a136"
server_key_version: "encryptionkey@0"
I20260928 10:58:16.994042 8733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 10:58:16.997265 8747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:16.998657 8733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:16.998804 8733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "2514964d10644f9789d756c377d69f5f"
format_stamp: "Formatted at 2026-09-28 10:58:16 on dist-test-slave-djjw"
server_key: "40149abe2066aeb9f1895a94c66ced29"
server_key_iv: "9fe263328dd61adaf5dbde76fea1a136"
server_key_version: "encryptionkey@0"
I20260928 10:58:16.999001 8733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:17.004273 8733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:17.004998 8733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:17.005255 8733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:17.020919 8733 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:41073
I20260928 10:58:17.020926 8798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:41073 every 8 connection(s)
I20260928 10:58:17.022313 8733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:17.025969 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:17.030303 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8733
I20260928 10:58:17.030473 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:17.031109 4081 external_mini_cluster.cc:1428] Setting key 6a3eb0940a4c8493dba370beec46c703
I20260928 10:58:17.037853 8799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f: Bootstrap starting.
I20260928 10:58:17.040807 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:17.033189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.3.252.126:41073)
0928 10:58:17.034360 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:17.034376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:17.034437 (+ 61us) server_negotiation.cc:407] Connection header received
0928 10:58:17.036708 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:17.036736 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:17.037294 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:17.037754 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:17.038786 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:17.038794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:17.038811 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:17.038834 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:17.038903 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:17.038937 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:17.039148 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:17.039270 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 10:58:17.039562 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":362,"threads_started":1}
I20260928 10:58:17.041664 8799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:17.042712 8799 log.cc:824] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:17.045372 8799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f: No bootstrap required, opened a new log
I20260928 10:58:17.052372 8799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2514964d10644f9789d756c377d69f5f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41073 } }
I20260928 10:58:17.052744 8799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:17.052830 8799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2514964d10644f9789d756c377d69f5f, State: Initialized, Role: FOLLOWER
I20260928 10:58:17.053359 8799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2514964d10644f9789d756c377d69f5f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41073 } }
I20260928 10:58:17.053582 8799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:17.053699 8799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:17.053884 8799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:17.055931 8799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2514964d10644f9789d756c377d69f5f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41073 } }
I20260928 10:58:17.056396 8799 leader_election.cc:302] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2514964d10644f9789d756c377d69f5f; no voters:
I20260928 10:58:17.057026 8799 leader_election.cc:288] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:17.057174 8804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:17.057873 8804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [term 1 LEADER]: Becoming Leader. State: Replica: 2514964d10644f9789d756c377d69f5f, State: Running, Role: LEADER
I20260928 10:58:17.058482 8804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2514964d10644f9789d756c377d69f5f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41073 } }
I20260928 10:58:17.059130 8799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:17.061106 8806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2514964d10644f9789d756c377d69f5f. Latest consensus state: current_term: 1 leader_uuid: "2514964d10644f9789d756c377d69f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2514964d10644f9789d756c377d69f5f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41073 } } }
I20260928 10:58:17.061431 8805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2514964d10644f9789d756c377d69f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2514964d10644f9789d756c377d69f5f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41073 } } }
I20260928 10:58:17.061698 8806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:17.061748 8805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:17.063705 8812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:17.069144 8812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:17.078485 8812 catalog_manager.cc:1347] Generated new cluster ID: 143458ab22ba4e79becc59405d7e0d5c
I20260928 10:58:17.078650 8812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:17.114799 8812 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:17.115235 8812 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:17.122053 8812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f: Generated new TSK 0
I20260928 10:58:17.122857 8812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:58:17.127933 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41073
--builtin_ntp_servers=127.3.252.84:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 10:58:17.444725 8823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:17.444978 8823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:17.445088 8823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:17.452183 8823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:17.452287 8823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:17.452426 8823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:17.461537 8823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:17.462877 8823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:17.465047 8823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:17.478029 8828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:17.478024 8831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:17.478171 8829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:17.478706 8823 server_base.cc:1034] running on GCE node
I20260928 10:58:17.479354 8823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:17.480325 8823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:17.481540 8823 hybrid_clock.cc:648] HybridClock initialized: now 1790593097481477 us; error 56 us; skew 500 ppm
I20260928 10:58:17.481871 8823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:17.484112 8823 webserver.cc:460] Webserver started at http://127.3.252.65:40609/ using document root <none> and password file <none>
I20260928 10:58:17.484802 8823 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:17.484897 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:17.485188 8823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:17.487998 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "9535f1bd967c49ce8a71278fce0e680a"
format_stamp: "Formatted at 2026-09-28 10:58:17 on dist-test-slave-djjw"
server_key: "d7c007308aa102d75ff36200407d29a6"
server_key_iv: "a59af671cdf3699a40c21bbcc9cd02b7"
server_key_version: "encryptionkey@0"
I20260928 10:58:17.488771 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "9535f1bd967c49ce8a71278fce0e680a"
format_stamp: "Formatted at 2026-09-28 10:58:17 on dist-test-slave-djjw"
server_key: "d7c007308aa102d75ff36200407d29a6"
server_key_iv: "a59af671cdf3699a40c21bbcc9cd02b7"
server_key_version: "encryptionkey@0"
I20260928 10:58:17.494130 8823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 10:58:17.497527 8837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:17.498965 8823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:17.499117 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "9535f1bd967c49ce8a71278fce0e680a"
format_stamp: "Formatted at 2026-09-28 10:58:17 on dist-test-slave-djjw"
server_key: "d7c007308aa102d75ff36200407d29a6"
server_key_iv: "a59af671cdf3699a40c21bbcc9cd02b7"
server_key_version: "encryptionkey@0"
I20260928 10:58:17.499326 8823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:17.504717 8823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:17.505491 8823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:17.505731 8823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:17.506595 8823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:17.507855 8823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:17.507936 8823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:17.508018 8823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:17.508085 8823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:17.536906 8823 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:38265
I20260928 10:58:17.536957 8949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:38265 every 8 connection(s)
I20260928 10:58:17.538157 8823 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:17.545589 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:17.540634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:56207 (local address 127.3.252.126:41073)
0928 10:58:17.540807 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:17.540811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:17.542148 (+ 1337us) server_negotiation.cc:407] Connection header received
0928 10:58:17.543744 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:17.543748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:17.543804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:17.543890 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:17.545072 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:17.545076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:17.545079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:17.545094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:17.545124 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:17.545135 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:17.545225 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:17.545348 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 10:58:17.545394 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 10:58:17.546164 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8823
I20260928 10:58:17.546347 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:17.546577 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:17.541076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:41073 (local address 127.3.252.65:56207)
0928 10:58:17.541996 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:17.542042 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:17.543496 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:17.544077 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:17.544084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:17.544174 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:17.544792 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:17.544794 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:17.544828 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:17.544908 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:17.545206 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:17.545211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:17.545216 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 10:58:17.545406 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 10:58:17.545593 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":261,"threads_started":1}
I20260928 10:58:17.546962 4081 external_mini_cluster.cc:1428] Setting key fdea2d1aa08b28fd75d9482a6a57038c
I20260928 10:58:17.547976 8950 heartbeater.cc:347] Connected to a master server at 127.3.252.126:41073
I20260928 10:58:17.548338 8950 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:17.548972 8950 heartbeater.cc:511] Master 127.3.252.126:41073 requested a full tablet report, sending...
I20260928 10:58:17.549890 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41073
--builtin_ntp_servers=127.3.252.84:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 10:58:17.551016 8764 ts_manager.cc:194] Registered new tserver with Master: 9535f1bd967c49ce8a71278fce0e680a (127.3.252.65:38265)
W20260928 10:58:17.858122 8953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:17.858393 8953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:17.858503 8953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:17.865679 8953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:17.865777 8953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:17.865935 8953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:17.874943 8953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:17.876199 8953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:17.877946 8953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:17.890221 8958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:17.890376 8959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:17.890410 8961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:17.891492 8953 server_base.cc:1034] running on GCE node
I20260928 10:58:17.892042 8953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:17.892872 8953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:17.894053 8953 hybrid_clock.cc:648] HybridClock initialized: now 1790593097894010 us; error 31 us; skew 500 ppm
I20260928 10:58:17.894327 8953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:17.896467 8953 webserver.cc:460] Webserver started at http://127.3.252.66:42973/ using document root <none> and password file <none>
I20260928 10:58:17.897100 8953 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:17.897178 8953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:17.897426 8953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:17.900138 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "cd989eb36ddd444cb0bc14a9b2c281f2"
format_stamp: "Formatted at 2026-09-28 10:58:17 on dist-test-slave-djjw"
server_key: "b04f3e5a20b0a1f0f225c1db5c1e9a8d"
server_key_iv: "748a5cbc9311d069695a627420daac4b"
server_key_version: "encryptionkey@0"
I20260928 10:58:17.900825 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "cd989eb36ddd444cb0bc14a9b2c281f2"
format_stamp: "Formatted at 2026-09-28 10:58:17 on dist-test-slave-djjw"
server_key: "b04f3e5a20b0a1f0f225c1db5c1e9a8d"
server_key_iv: "748a5cbc9311d069695a627420daac4b"
server_key_version: "encryptionkey@0"
I20260928 10:58:17.905647 8953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 10:58:17.908843 8967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:17.910136 8953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:17.910261 8953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "cd989eb36ddd444cb0bc14a9b2c281f2"
format_stamp: "Formatted at 2026-09-28 10:58:17 on dist-test-slave-djjw"
server_key: "b04f3e5a20b0a1f0f225c1db5c1e9a8d"
server_key_iv: "748a5cbc9311d069695a627420daac4b"
server_key_version: "encryptionkey@0"
I20260928 10:58:17.910441 8953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:17.915582 8953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:17.916297 8953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:17.916508 8953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:17.917352 8953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:17.918726 8953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:17.918814 8953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:17.918892 8953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:17.918942 8953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:17.947278 8953 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:37375
I20260928 10:58:17.947346 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:37375 every 8 connection(s)
I20260928 10:58:17.948506 8953 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:17.956005 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:17.950861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:57899 (local address 127.3.252.126:41073)
0928 10:58:17.951010 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:17.951014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:17.952366 (+ 1352us) server_negotiation.cc:407] Connection header received
0928 10:58:17.953939 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:17.953945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:17.954014 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:17.954098 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:17.955306 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:17.955311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:17.955314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:17.955330 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:17.955362 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:17.955373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:17.955490 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:17.955665 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 10:58:17.955729 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:58:17.956866 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:17.951344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:41073 (local address 127.3.252.66:57899)
0928 10:58:17.952218 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:17.952251 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:17.953677 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:17.954321 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:17.954329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:17.954418 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:17.954998 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:17.955001 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:17.955038 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:17.955120 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:17.955462 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:17.955472 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:17.955479 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:58:17.955733 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 10:58:17.955989 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":252,"threads_started":1}
I20260928 10:58:17.957619 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 8953
I20260928 10:58:17.957772 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:17.958210 9080 heartbeater.cc:347] Connected to a master server at 127.3.252.126:41073
I20260928 10:58:17.958321 4081 external_mini_cluster.cc:1428] Setting key 9a6514700a9a8bdad80febf17634b0a7
I20260928 10:58:17.958493 9080 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:17.959074 9080 heartbeater.cc:511] Master 127.3.252.126:41073 requested a full tablet report, sending...
I20260928 10:58:17.960316 8764 ts_manager.cc:194] Registered new tserver with Master: cd989eb36ddd444cb0bc14a9b2c281f2 (127.3.252.66:37375)
I20260928 10:58:17.960788 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41073
--builtin_ntp_servers=127.3.252.84:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 10:58:18.280012 9083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:18.280288 9083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:18.280398 9083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:18.287791 9083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:18.287894 9083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:18.288028 9083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:18.297242 9083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:41073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:18.298599 9083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:18.300379 9083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:18.312918 9089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:18.312947 9091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:18.312918 9088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:18.313982 9083 server_base.cc:1034] running on GCE node
I20260928 10:58:18.314600 9083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:18.315501 9083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:18.316705 9083 hybrid_clock.cc:648] HybridClock initialized: now 1790593098316644 us; error 48 us; skew 500 ppm
I20260928 10:58:18.316999 9083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:18.319217 9083 webserver.cc:460] Webserver started at http://127.3.252.67:41791/ using document root <none> and password file <none>
I20260928 10:58:18.319881 9083 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:18.319968 9083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:18.320255 9083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:18.323243 9083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "d5767a46e9ff4b10a96a3d9587c7e357"
format_stamp: "Formatted at 2026-09-28 10:58:18 on dist-test-slave-djjw"
server_key: "e8de08bc4f1033b11b2e118b5356ae56"
server_key_iv: "456f95cdf3821b4bd41ae2739e27a857"
server_key_version: "encryptionkey@0"
I20260928 10:58:18.324020 9083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "d5767a46e9ff4b10a96a3d9587c7e357"
format_stamp: "Formatted at 2026-09-28 10:58:18 on dist-test-slave-djjw"
server_key: "e8de08bc4f1033b11b2e118b5356ae56"
server_key_iv: "456f95cdf3821b4bd41ae2739e27a857"
server_key_version: "encryptionkey@0"
I20260928 10:58:18.328954 9083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:58:18.332428 9097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:18.334002 9083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:18.334152 9083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "d5767a46e9ff4b10a96a3d9587c7e357"
format_stamp: "Formatted at 2026-09-28 10:58:18 on dist-test-slave-djjw"
server_key: "e8de08bc4f1033b11b2e118b5356ae56"
server_key_iv: "456f95cdf3821b4bd41ae2739e27a857"
server_key_version: "encryptionkey@0"
I20260928 10:58:18.334367 9083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:18.339641 9083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:18.340425 9083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:18.340655 9083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:18.341574 9083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:18.342885 9083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:18.342969 9083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:18.343072 9083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:18.343137 9083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:18.371366 9083 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:45385
I20260928 10:58:18.371425 9209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:45385 every 8 connection(s)
I20260928 10:58:18.372571 9083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:18.380173 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.374885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:47721 (local address 127.3.252.126:41073)
0928 10:58:18.375127 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.375131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.376493 (+ 1362us) server_negotiation.cc:407] Connection header received
0928 10:58:18.378135 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.378141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.378207 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.378295 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.379480 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.379486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.379491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.379511 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.379541 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.379552 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.379659 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.379826 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.379892 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 10:58:18.381110 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.375436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:41073 (local address 127.3.252.67:47721)
0928 10:58:18.376310 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.376350 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:18.377867 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:18.378528 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:18.378536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:18.378629 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:18.379199 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:18.379202 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:18.379237 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:18.379318 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:18.379639 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:18.379648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:18.379656 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:58:18.379893 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 10:58:18.380144 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":262,"threads_started":1}
I20260928 10:58:18.382385 9210 heartbeater.cc:347] Connected to a master server at 127.3.252.126:41073
I20260928 10:58:18.382529 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9083
I20260928 10:58:18.382673 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:18.382684 9210 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:18.383069 4081 external_mini_cluster.cc:1428] Setting key c2f42296653a199b31043ba1797c847c
I20260928 10:58:18.383288 9210 heartbeater.cc:511] Master 127.3.252.126:41073 requested a full tablet report, sending...
I20260928 10:58:18.384624 8764 ts_manager.cc:194] Registered new tserver with Master: d5767a46e9ff4b10a96a3d9587c7e357 (127.3.252.67:45385)
I20260928 10:58:18.385962 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:58:18.392557 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.390229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.3.252.126:41073)
0928 10:58:18.390402 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.390407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.390879 (+ 472us) server_negotiation.cc:407] Connection header received
0928 10:58:18.391022 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.391025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.391078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.391161 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.392013 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.392017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.392021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.392036 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.392067 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.392078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.392176 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.392289 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.392335 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:58:18.396592 8763 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:58:18.403307 8763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:18.418993 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.414527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:45385 (local address 127.0.0.1:49610)
0928 10:58:18.415466 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.415507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:18.415654 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:18.416664 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:18.416673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:18.416776 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:18.417603 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:18.417606 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:18.417646 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:18.417723 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:18.418174 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:18.418182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:18.418189 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:58:18.418350 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 10:58:18.418442 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":513,"threads_started":1}
I20260928 10:58:18.419327 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.414794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.3.252.67:45385)
0928 10:58:18.416192 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.416200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.416233 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:58:18.416300 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.416307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.416491 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.416603 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.417924 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.417933 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.417950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.417969 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.418029 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.418053 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.418812 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.418888 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.418981 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":1097,"threads_started":1}
I20260928 10:58:18.423704 9145 tablet_service.cc:1467] Processing CreateTablet for tablet 2b505e297e0240dab00c199b50503033 (DEFAULT_TABLE table=test-table [id=f8e9657569d14e4ba2a5020944be31e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:18.425447 9145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b505e297e0240dab00c199b50503033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:18.427790 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.414856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:35974)
0928 10:58:18.417754 (+ 2898us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.417772 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:18.417899 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:18.426061 (+ 8162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:18.426066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:18.426090 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:18.426657 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:18.426660 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:18.426678 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:18.426723 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:18.427208 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:18.427213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:18.427218 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 10:58:18.427339 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 10:58:18.427395 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2787,"thread_start_us":2645,"threads_started":1}
I20260928 10:58:18.428614 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.415415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.3.252.65:38265)
0928 10:58:18.425433 (+ 10018us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.425441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.425477 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:58:18.425537 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.425543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.425727 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.425893 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.426966 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.426973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.426989 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.427005 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.427061 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.427086 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.428163 (+ 1077us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.428227 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.428325 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9874,"thread_start_us":9773,"threads_started":1}
I20260928 10:58:18.429055 9227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.418219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.3.252.66:37375)
0928 10:58:18.425168 (+ 6949us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.425177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.425209 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:58:18.425278 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.425285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.425513 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.427405 (+ 1892us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.428324 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.428331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.428342 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.428366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.428411 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.428427 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.428586 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.428656 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.428745 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6793,"thread_start_us":6668,"threads_started":1}
I20260928 10:58:18.429147 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.417870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:37375 (local address 127.0.0.1:40474)
0928 10:58:18.419421 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.419437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:18.420523 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:18.425703 (+ 5180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:18.425708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:18.425733 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:18.427202 (+ 1469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:18.427205 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:18.427221 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:18.427259 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:18.428537 (+ 1278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:18.428542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:18.428548 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:58:18.428667 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 10:58:18.428716 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":225,"threads_started":1}
I20260928 10:58:18.432588 9015 tablet_service.cc:1467] Processing CreateTablet for tablet 2b505e297e0240dab00c199b50503033 (DEFAULT_TABLE table=test-table [id=f8e9657569d14e4ba2a5020944be31e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:18.432588 8885 tablet_service.cc:1467] Processing CreateTablet for tablet 2b505e297e0240dab00c199b50503033 (DEFAULT_TABLE table=test-table [id=f8e9657569d14e4ba2a5020944be31e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:18.433995 8885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b505e297e0240dab00c199b50503033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:18.433995 9015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b505e297e0240dab00c199b50503033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:18.437212 9228 tablet_bootstrap.cc:492] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Bootstrap starting.
I20260928 10:58:18.440744 9228 tablet_bootstrap.cc:654] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:18.441694 9228 log.cc:824] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:18.444139 9228 tablet_bootstrap.cc:492] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: No bootstrap required, opened a new log
I20260928 10:58:18.444376 9228 ts_tablet_manager.cc:1397] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 10:58:18.448426 9231 tablet_bootstrap.cc:492] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: Bootstrap starting.
I20260928 10:58:18.449170 9230 tablet_bootstrap.cc:492] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Bootstrap starting.
I20260928 10:58:18.452246 9231 tablet_bootstrap.cc:654] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:18.452661 9230 tablet_bootstrap.cc:654] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:18.453281 9231 log.cc:824] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:18.453236 9228 raft_consensus.cc:348] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.453688 9228 raft_consensus.cc:374] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:18.453783 9230 log.cc:824] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:18.453888 9228 raft_consensus.cc:729] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5767a46e9ff4b10a96a3d9587c7e357, State: Initialized, Role: FOLLOWER
I20260928 10:58:18.454681 9228 consensus_queue.cc:260] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.456118 9231 tablet_bootstrap.cc:492] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: No bootstrap required, opened a new log
I20260928 10:58:18.456362 9231 ts_tablet_manager.cc:1397] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260928 10:58:18.458559 9210 heartbeater.cc:503] Master 127.3.252.126:41073 was elected leader, sending a full tablet report...
I20260928 10:58:18.459285 9228 ts_tablet_manager.cc:1428] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260928 10:58:18.462163 9230 tablet_bootstrap.cc:492] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: No bootstrap required, opened a new log
I20260928 10:58:18.462441 9230 ts_tablet_manager.cc:1397] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260928 10:58:18.465937 9231 raft_consensus.cc:348] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.466385 9231 raft_consensus.cc:374] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:18.466482 9231 raft_consensus.cc:729] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9535f1bd967c49ce8a71278fce0e680a, State: Initialized, Role: FOLLOWER
I20260928 10:58:18.467113 9231 consensus_queue.cc:260] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.468117 8950 heartbeater.cc:503] Master 127.3.252.126:41073 was elected leader, sending a full tablet report...
I20260928 10:58:18.468811 9231 ts_tablet_manager.cc:1428] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260928 10:58:18.471454 9230 raft_consensus.cc:348] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.471791 9230 raft_consensus.cc:374] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:18.471891 9230 raft_consensus.cc:729] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd989eb36ddd444cb0bc14a9b2c281f2, State: Initialized, Role: FOLLOWER
I20260928 10:58:18.472424 9230 consensus_queue.cc:260] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.473285 9080 heartbeater.cc:503] Master 127.3.252.126:41073 was elected leader, sending a full tablet report...
I20260928 10:58:18.473913 9230 ts_tablet_manager.cc:1428] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20260928 10:58:18.540316 8951 tablet.cc:2369] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:58:18.624118 9211 tablet.cc:2369] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:58:18.641306 9234 raft_consensus.cc:482] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:18.641587 9234 raft_consensus.cc:504] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.643177 9234 leader_election.cc:288] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9535f1bd967c49ce8a71278fce0e680a (127.3.252.65:38265), cd989eb36ddd444cb0bc14a9b2c281f2 (127.3.252.66:37375)
I20260928 10:58:18.645658 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.643641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:37871 (local address 127.3.252.65:38265)
0928 10:58:18.643798 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.643803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.643857 (+ 54us) server_negotiation.cc:407] Connection header received
0928 10:58:18.643922 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.643926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.644002 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.644088 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.644975 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.644981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.644985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.645003 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.645039 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.645055 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.645165 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.645328 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.645389 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:58:18.645876 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.643484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.3.252.67:37871)
0928 10:58:18.643656 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.643677 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:18.643779 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:18.644138 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:18.644144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:18.644170 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:18.644766 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:18.644769 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:18.644785 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:18.644828 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:18.645190 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:18.645195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:18.645200 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 10:58:18.645316 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 10:58:18.645630 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 10:58:18.646775 9227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.644143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:38501 (local address 127.3.252.66:37375)
0928 10:58:18.644323 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.644329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.644738 (+ 409us) server_negotiation.cc:407] Connection header received
0928 10:58:18.644856 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.644859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.644906 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.644990 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.646021 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.646026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.646030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.646077 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.646118 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.646156 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.646268 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.646494 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.646554 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:58:18.647378 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.643939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:37375 (local address 127.3.252.67:38501)
0928 10:58:18.644623 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.644639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:18.644754 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:18.645007 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:18.645011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:18.645031 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:18.645667 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:58:18.645671 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:18.645685 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:58:18.645755 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:18.646355 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:18.646359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:18.646363 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 10:58:18.646964 (+ 601us) client_negotiation.cc:240] Negotiation successful
0928 10:58:18.647015 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260928 10:58:18.647334 8905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b505e297e0240dab00c199b50503033" candidate_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535f1bd967c49ce8a71278fce0e680a" is_pre_election: true
I20260928 10:58:18.647917 8905 raft_consensus.cc:2454] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5767a46e9ff4b10a96a3d9587c7e357 in term 0.
I20260928 10:58:18.648630 9101 leader_election.cc:302] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9535f1bd967c49ce8a71278fce0e680a, d5767a46e9ff4b10a96a3d9587c7e357; no voters:
I20260928 10:58:18.648768 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b505e297e0240dab00c199b50503033" candidate_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" is_pre_election: true
I20260928 10:58:18.648957 9234 raft_consensus.cc:2781] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:18.649091 9234 raft_consensus.cc:482] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:18.649180 9234 raft_consensus.cc:3037] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:18.649196 9035 raft_consensus.cc:2454] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5767a46e9ff4b10a96a3d9587c7e357 in term 0.
I20260928 10:58:18.651621 9234 raft_consensus.cc:504] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.652144 9234 leader_election.cc:288] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [CANDIDATE]: Term 1 election: Requested vote from peers 9535f1bd967c49ce8a71278fce0e680a (127.3.252.65:38265), cd989eb36ddd444cb0bc14a9b2c281f2 (127.3.252.66:37375)
I20260928 10:58:18.652503 8905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b505e297e0240dab00c199b50503033" candidate_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535f1bd967c49ce8a71278fce0e680a"
I20260928 10:58:18.652576 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b505e297e0240dab00c199b50503033" candidate_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2"
I20260928 10:58:18.652694 8905 raft_consensus.cc:3037] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:18.652756 9035 raft_consensus.cc:3037] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:18.654850 8905 raft_consensus.cc:2454] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5767a46e9ff4b10a96a3d9587c7e357 in term 1.
I20260928 10:58:18.654949 9035 raft_consensus.cc:2454] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5767a46e9ff4b10a96a3d9587c7e357 in term 1.
I20260928 10:58:18.655210 9101 leader_election.cc:302] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9535f1bd967c49ce8a71278fce0e680a, d5767a46e9ff4b10a96a3d9587c7e357; no voters:
I20260928 10:58:18.655483 9234 raft_consensus.cc:2781] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:18.655880 9234 raft_consensus.cc:686] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 1 LEADER]: Becoming Leader. State: Replica: d5767a46e9ff4b10a96a3d9587c7e357, State: Running, Role: LEADER
I20260928 10:58:18.656383 9234 consensus_queue.cc:237] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } }
I20260928 10:58:18.660755 8763 catalog_manager.cc:5495] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5767a46e9ff4b10a96a3d9587c7e357 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5767a46e9ff4b10a96a3d9587c7e357" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 45385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 } health_report { overall_health: UNKNOWN } } }
W20260928 10:58:18.700351 9081 tablet.cc:2369] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:58:18.716533 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.714570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.3.252.67:45385)
0928 10:58:18.714739 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.714745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.714763 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:58:18.714979 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.714984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.715064 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.715150 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.715985 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.715991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.715996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.716012 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.716044 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.716055 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.716160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.716241 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.716298 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:58:18.724423 9035 raft_consensus.cc:1261] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 1 FOLLOWER]: Refusing update from remote peer d5767a46e9ff4b10a96a3d9587c7e357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:18.724426 8905 raft_consensus.cc:1261] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a [term 1 FOLLOWER]: Refusing update from remote peer d5767a46e9ff4b10a96a3d9587c7e357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:18.725273 9234 consensus_queue.cc:1035] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd989eb36ddd444cb0bc14a9b2c281f2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:18.725564 9238 consensus_queue.cc:1035] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9535f1bd967c49ce8a71278fce0e680a" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:18.738063 9243 mvcc.cc:205] Tried to move back new op lower bound from 7334269332365852672 to 7334269332100349952. Current Snapshot: MvccSnapshot[applied={T|T < 7334269332365852672}]
I20260928 10:58:18.749188 8763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:18.749468 8763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:18.752977 8763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2514964d10644f9789d756c377d69f5f: Sending DeleteTablet for 3 replicas of tablet 2b505e297e0240dab00c199b50503033
I20260928 10:58:18.754115 8885 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b505e297e0240dab00c199b50503033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:18 UTC) from {username='slave'} at 127.0.0.1:35974
I20260928 10:58:18.754216 9015 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b505e297e0240dab00c199b50503033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:18 UTC) from {username='slave'} at 127.0.0.1:40474
I20260928 10:58:18.754799 9145 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b505e297e0240dab00c199b50503033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:18 UTC) from {username='slave'} at 127.0.0.1:49610
I20260928 10:58:18.756242 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8823
I20260928 10:58:18.783370 9253 tablet_replica.cc:330] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: stopping tablet replica
I20260928 10:58:18.785250 9253 raft_consensus.cc:2229] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:18.786571 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.757760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.3.252.65:38265)
0928 10:58:18.757968 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.757976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.781742 (+ 23766us) server_negotiation.cc:407] Connection header received
0928 10:58:18.782389 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.782396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.782539 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.782620 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.785626 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.785634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.785640 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.785669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.785715 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.785733 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.785890 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.786138 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.786212 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:58:18.788098 8945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:18.788527 9253 raft_consensus.cc:2258] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:18.791244 9253 ts_tablet_manager.cc:1905] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:18.795272 9252 tablet_replica.cc:330] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: stopping tablet replica
I20260928 10:58:18.795790 9252 raft_consensus.cc:2229] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:18.796546 9252 raft_consensus.cc:2258] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:18.798908 9252 ts_tablet_manager.cc:1905] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:18.800086 9253 ts_tablet_manager.cc:1918] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:18.800287 9253 log.cc:1196] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/2b505e297e0240dab00c199b50503033
I20260928 10:58:18.800868 9253 ts_tablet_manager.cc:1939] T 2b505e297e0240dab00c199b50503033 P d5767a46e9ff4b10a96a3d9587c7e357: Deleting consensus metadata
I20260928 10:58:18.802712 8748 catalog_manager.cc:4841] TS d5767a46e9ff4b10a96a3d9587c7e357 (127.3.252.67:45385): tablet 2b505e297e0240dab00c199b50503033 (table test-table [id=f8e9657569d14e4ba2a5020944be31e4]) successfully deleted
I20260928 10:58:18.806181 9252 ts_tablet_manager.cc:1918] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:18.806326 9252 log.cc:1196] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/2b505e297e0240dab00c199b50503033
I20260928 10:58:18.806789 9252 ts_tablet_manager.cc:1939] T 2b505e297e0240dab00c199b50503033 P cd989eb36ddd444cb0bc14a9b2c281f2: Deleting consensus metadata
I20260928 10:58:18.808485 8751 catalog_manager.cc:4841] TS cd989eb36ddd444cb0bc14a9b2c281f2 (127.3.252.66:37375): tablet 2b505e297e0240dab00c199b50503033 (table test-table [id=f8e9657569d14e4ba2a5020944be31e4]) successfully deleted
I20260928 10:58:18.879787 9251 tablet_replica.cc:330] T 2b505e297e0240dab00c199b50503033 P 9535f1bd967c49ce8a71278fce0e680a: stopping tablet replica
I20260928 10:58:18.879792 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8823
W20260928 10:58:18.894342 8751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.252.65:38265 (error 108)
I20260928 10:58:18.894951 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8953
I20260928 10:58:18.896242 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.895714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:36000)
0928 10:58:18.895952 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.896078 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260928 10:58:18.896512 8751 catalog_manager.cc:4584] TS 9535f1bd967c49ce8a71278fce0e680a (127.3.252.65:38265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b505e297e0240dab00c199b50503033: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
I20260928 10:58:18.898046 9227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.895691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.3.252.66:37375)
0928 10:58:18.895867 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:18.895874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:18.895969 (+ 95us) server_negotiation.cc:407] Connection header received
0928 10:58:18.896168 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:18.896174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:18.896284 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:18.896389 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:18.897397 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:18.897404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:18.897409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:18.897428 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:18.897467 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:18.897483 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:18.897609 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:18.897677 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 10:58:18.897735 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:58:18.898669 9075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:18.950639 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.950225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:36010)
0928 10:58:18.950416 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.950499 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 10:58:18.951602 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:18.951291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:36022)
0928 10:58:18.951424 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:18.951493 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 10:58:19.012231 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:19.011719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:36030)
0928 10:58:19.011928 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:19.012042 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 10:58:19.013355 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:19.012969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:36032)
0928 10:58:19.013127 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:19.013213 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 10:58:19.032256 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8953
I20260928 10:58:19.052014 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9083
I20260928 10:58:19.054927 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:19.052820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.3.252.67:45385)
0928 10:58:19.053046 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:19.053052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:19.053084 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:58:19.053250 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:19.053256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:19.053323 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:19.053407 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:19.054315 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:19.054321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:19.054326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:19.054343 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:19.054371 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:19.054382 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:19.054511 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:19.054572 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 10:58:19.054635 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 10:58:19.055543 9205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:19.093963 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:19.093437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:36044)
0928 10:58:19.093655 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:19.093755 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 10:58:19.095005 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:19.094629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38265 (local address 127.0.0.1:36048)
0928 10:58:19.094809 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:19.094892 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 10:58:19.171012 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9083
I20260928 10:58:19.186089 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 8733
I20260928 10:58:19.187242 8794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:19.290074 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 8733
2026-09-28T10:58:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2715 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 10:58:19.326992 4081 mini_oidc.cc:152] Starting JWKS server
I20260928 10:58:19.329478 4081 webserver.cc:460] Webserver started at https://127.0.0.1:42303/ using document root <none> and password file <none>
I20260928 10:58:19.329749 4081 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:58:19.330447 4081 webserver.cc:460] Webserver started at http://127.0.0.1:36843/ using document root <none> and password file <none>
2026-09-28T10:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:19Z Disabled control of system clock
I20260928 10:58:19.354574 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:34991
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:34991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42303/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/testchainca.cert with env {}
W20260928 10:58:19.667569 9265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:19.667863 9265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:19.667990 9265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:19.675179 9265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:19.675295 9265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:19.675360 9265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:19.675439 9265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:19.675501 9265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:58:19.675577 9265 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:58:19.675628 9265 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42303/jwks.json
I20260928 10:58:19.685468 9265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:34991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42303/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:19.686749 9265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:19.687950 9265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:19.699257 9270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:19.699219 9271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:19.699180 9273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:19.700601 9265 server_base.cc:1034] running on GCE node
I20260928 10:58:19.701354 9265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:19.703001 9265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:19.704195 9265 hybrid_clock.cc:648] HybridClock initialized: now 1790593099704156 us; error 34 us; skew 500 ppm
I20260928 10:58:19.704535 9265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:19.710428 9265 webserver.cc:460] Webserver started at http://127.3.252.126:35163/ using document root <none> and password file <none>
I20260928 10:58:19.711146 9265 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:19.711251 9265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:19.711544 9265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:19.714470 9265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "a766bb23b62a46f695edf12b0e23bd39"
format_stamp: "Formatted at 2026-09-28 10:58:19 on dist-test-slave-djjw"
server_key: "cf0e07ac639cc9d89944a9bf652bcf6c"
server_key_iv: "f5b94e81c5b35c21fd39631c40c5e0aa"
server_key_version: "encryptionkey@0"
I20260928 10:58:19.715246 9265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "a766bb23b62a46f695edf12b0e23bd39"
format_stamp: "Formatted at 2026-09-28 10:58:19 on dist-test-slave-djjw"
server_key: "cf0e07ac639cc9d89944a9bf652bcf6c"
server_key_iv: "f5b94e81c5b35c21fd39631c40c5e0aa"
server_key_version: "encryptionkey@0"
I20260928 10:58:19.720168 9265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:19.723489 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:19.724889 9265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:58:19.725067 9265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "a766bb23b62a46f695edf12b0e23bd39"
format_stamp: "Formatted at 2026-09-28 10:58:19 on dist-test-slave-djjw"
server_key: "cf0e07ac639cc9d89944a9bf652bcf6c"
server_key_iv: "f5b94e81c5b35c21fd39631c40c5e0aa"
server_key_version: "encryptionkey@0"
I20260928 10:58:19.725270 9265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:19.752851 9265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:19.753666 9265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:19.753985 9265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:19.770251 9265 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:34991
I20260928 10:58:19.770270 9330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:34991 every 8 connection(s)
I20260928 10:58:19.771553 9265 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:19.775127 9331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:19.781349 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9265
I20260928 10:58:19.781570 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:19.782382 4081 external_mini_cluster.cc:1428] Setting key e5242d8649b6e3f2b36e83954f01e546
I20260928 10:58:19.786012 9331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39: Bootstrap starting.
I20260928 10:58:19.788980 9331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:19.790019 9331 log.cc:824] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:19.792676 9331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39: No bootstrap required, opened a new log
I20260928 10:58:19.795735 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:19.784290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.3.252.126:34991)
0928 10:58:19.785273 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:19.785284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:19.785370 (+ 86us) server_negotiation.cc:407] Connection header received
0928 10:58:19.787135 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:19.787156 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:19.787629 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:19.787984 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:19.788568 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:19.789907 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:19.791549 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:19.792191 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:19.792367 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:19.793488 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:19.793495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:19.793509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:19.794065 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:19.794155 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:19.794175 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:19.794291 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:19.794495 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 10:58:19.794779 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":322,"threads_started":1}
I20260928 10:58:19.799511 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:58:19.800688 9331 raft_consensus.cc:348] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a766bb23b62a46f695edf12b0e23bd39" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34991 } }
I20260928 10:58:19.801007 9331 raft_consensus.cc:374] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:19.801088 9331 raft_consensus.cc:729] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a766bb23b62a46f695edf12b0e23bd39, State: Initialized, Role: FOLLOWER
I20260928 10:58:19.801680 9331 consensus_queue.cc:260] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a766bb23b62a46f695edf12b0e23bd39" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34991 } }
I20260928 10:58:19.801997 9331 raft_consensus.cc:388] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:19.802150 9331 raft_consensus.cc:482] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:19.802336 9331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:19.804555 9331 raft_consensus.cc:504] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a766bb23b62a46f695edf12b0e23bd39" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34991 } }
I20260928 10:58:19.804982 9331 leader_election.cc:302] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a766bb23b62a46f695edf12b0e23bd39; no voters:
I20260928 10:58:19.805488 9331 leader_election.cc:288] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:19.805609 9336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:19.806288 9336 raft_consensus.cc:686] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [term 1 LEADER]: Becoming Leader. State: Replica: a766bb23b62a46f695edf12b0e23bd39, State: Running, Role: LEADER
I20260928 10:58:19.806777 9336 consensus_queue.cc:237] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a766bb23b62a46f695edf12b0e23bd39" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34991 } }
I20260928 10:58:19.807312 9331 sys_catalog.cc:564] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:19.809262 9338 sys_catalog.cc:455] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a766bb23b62a46f695edf12b0e23bd39. Latest consensus state: current_term: 1 leader_uuid: "a766bb23b62a46f695edf12b0e23bd39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a766bb23b62a46f695edf12b0e23bd39" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34991 } } }
I20260928 10:58:19.809556 9338 sys_catalog.cc:458] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:19.810366 9341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:19.810237 9337 sys_catalog.cc:455] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a766bb23b62a46f695edf12b0e23bd39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a766bb23b62a46f695edf12b0e23bd39" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 34991 } } }
I20260928 10:58:19.810556 9337 sys_catalog.cc:458] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:19.815965 9341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:19.824723 9341 catalog_manager.cc:1347] Generated new cluster ID: 093bd3c4f67e4d859f626bb6def7465d
I20260928 10:58:19.824847 9341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:19.866946 9341 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:19.868507 9341 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:19.874259 9341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a766bb23b62a46f695edf12b0e23bd39: Generated new TSK 0
I20260928 10:58:19.875007 9341 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 10:58:19.914409 9361 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 10:58:19.914561 9361 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:58:19.915712 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:19.894371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.3.252.126:34991)
0928 10:58:19.894561 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:19.894568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:19.894995 (+ 427us) server_negotiation.cc:407] Connection header received
0928 10:58:19.895272 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:19.895278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:19.895382 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:19.895530 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:58:19.895958 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:19.897057 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:19.898465 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:19.899249 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:19.899383 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:19.899828 (+ 445us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:58:19.915066 (+ 15238us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 10:58:19.915433 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 10:58:19.915829 9334 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 10:58:19.916961 9360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:19.894292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:34991 (local address 127.0.0.1:45674)
0928 10:58:19.894907 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:19.894923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:19.895070 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:19.895541 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:19.895544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:19.895553 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:58:19.895830 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:19.895837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:19.897193 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:19.897197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:19.898322 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:19.898334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:19.899387 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:19.899391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:19.899516 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:19.899571 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:58:19.915336 (+ 15765us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 10:58:19.915600 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.252.126:34991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":303,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.252.126:34991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 10:58:19.918932 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9265
I20260928 10:58:19.920027 9326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:20.012761 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9265
2026-09-28T10:58:20Z chronyd exiting
I20260928 10:58:20.059798 4081 test_util.cc:175] -----------------------------------------------
I20260928 10:58:20.059930 4081 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790593056029081-4081-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (737 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 10:58:20.063613 4081 mini_oidc.cc:152] Starting JWKS server
I20260928 10:58:20.065392 4081 webserver.cc:460] Webserver started at https://127.0.0.1:34629/ using document root <none> and password file <none>
I20260928 10:58:20.065630 4081 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:58:20.066252 4081 webserver.cc:460] Webserver started at http://127.0.0.1:36513/ using document root <none> and password file <none>
2026-09-28T10:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:20Z Disabled control of system clock
I20260928 10:58:20.089921 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41903
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:46247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:41903
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34629/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/testchainca.cert with env {}
W20260928 10:58:20.414011 9368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:20.414345 9368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:20.414436 9368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:20.422305 9368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:20.422493 9368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:20.422538 9368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:20.422596 9368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:20.422650 9368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:58:20.422739 9368 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:58:20.422787 9368 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34629/jwks.json
I20260928 10:58:20.434651 9368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:46247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:41903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34629/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:20.436115 9368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:20.437537 9368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:20.449474 9373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:20.449558 9374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:20.449577 9376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:20.449792 9368 server_base.cc:1034] running on GCE node
I20260928 10:58:20.450891 9368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:20.452351 9368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:20.453552 9368 hybrid_clock.cc:648] HybridClock initialized: now 1790593100453503 us; error 51 us; skew 500 ppm
I20260928 10:58:20.453933 9368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:20.456288 9368 webserver.cc:460] Webserver started at http://127.3.252.126:36115/ using document root <none> and password file <none>
I20260928 10:58:20.456965 9368 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:20.457069 9368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:20.457376 9368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:20.460223 9368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "98810c9056bf49c38c6122163bc29684"
format_stamp: "Formatted at 2026-09-28 10:58:20 on dist-test-slave-djjw"
server_key: "706106ec9e5fec256914098f26625bb7"
server_key_iv: "085c61572d6a7b1c727bd5bc304e7e79"
server_key_version: "encryptionkey@0"
I20260928 10:58:20.460953 9368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "98810c9056bf49c38c6122163bc29684"
format_stamp: "Formatted at 2026-09-28 10:58:20 on dist-test-slave-djjw"
server_key: "706106ec9e5fec256914098f26625bb7"
server_key_iv: "085c61572d6a7b1c727bd5bc304e7e79"
server_key_version: "encryptionkey@0"
I20260928 10:58:20.465866 9368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:58:20.469060 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:20.470422 9368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 10:58:20.470568 9368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "98810c9056bf49c38c6122163bc29684"
format_stamp: "Formatted at 2026-09-28 10:58:20 on dist-test-slave-djjw"
server_key: "706106ec9e5fec256914098f26625bb7"
server_key_iv: "085c61572d6a7b1c727bd5bc304e7e79"
server_key_version: "encryptionkey@0"
I20260928 10:58:20.470752 9368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:20.482066 9368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:20.482801 9368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:20.483083 9368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:20.499411 9368 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:41903
I20260928 10:58:20.499429 9433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:41903 every 8 connection(s)
I20260928 10:58:20.500732 9368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:20.504307 9434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:20.508417 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9368
I20260928 10:58:20.508613 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:20.509215 4081 external_mini_cluster.cc:1428] Setting key 5a4b2cc6b475c60f433e23a50c48719d
I20260928 10:58:20.514585 9434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684: Bootstrap starting.
I20260928 10:58:20.517944 9434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:20.519052 9434 log.cc:824] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:20.522094 9434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684: No bootstrap required, opened a new log
I20260928 10:58:20.522333 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:20.511019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.3.252.126:41903)
0928 10:58:20.511971 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:20.511995 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:20.512038 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:58:20.513641 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:20.513668 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:20.514172 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:20.514489 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:20.515059 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:20.516306 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:20.518037 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:20.518710 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:20.518845 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:20.520014 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:20.520025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:20.520043 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:20.520656 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:20.520729 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:20.520749 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:20.520837 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:20.521057 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 10:58:20.521348 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":343,"threads_started":1}
I20260928 10:58:20.525766 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:58:20.529645 9434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98810c9056bf49c38c6122163bc29684" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41903 } }
I20260928 10:58:20.530014 9434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:20.530112 9434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98810c9056bf49c38c6122163bc29684, State: Initialized, Role: FOLLOWER
I20260928 10:58:20.530618 9434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98810c9056bf49c38c6122163bc29684" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41903 } }
I20260928 10:58:20.530812 9434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:20.530926 9434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:20.531067 9434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:20.533133 9434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98810c9056bf49c38c6122163bc29684" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41903 } }
I20260928 10:58:20.533587 9434 leader_election.cc:302] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98810c9056bf49c38c6122163bc29684; no voters:
I20260928 10:58:20.534166 9434 leader_election.cc:288] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:20.534253 9439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:20.535284 9439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [term 1 LEADER]: Becoming Leader. State: Replica: 98810c9056bf49c38c6122163bc29684, State: Running, Role: LEADER
I20260928 10:58:20.535761 9439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98810c9056bf49c38c6122163bc29684" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41903 } }
I20260928 10:58:20.536366 9434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:20.538290 9440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98810c9056bf49c38c6122163bc29684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98810c9056bf49c38c6122163bc29684" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41903 } } }
I20260928 10:58:20.538707 9440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:20.539366 9441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98810c9056bf49c38c6122163bc29684. Latest consensus state: current_term: 1 leader_uuid: "98810c9056bf49c38c6122163bc29684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98810c9056bf49c38c6122163bc29684" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41903 } } }
I20260928 10:58:20.539675 9441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:20.540933 9448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:20.545758 9448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:20.553157 9448 catalog_manager.cc:1347] Generated new cluster ID: 8aa0c8180e464ee5b3a696c3ae9dc710
I20260928 10:58:20.553263 9448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:20.565093 9448 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:20.566381 9448 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:20.575076 9448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98810c9056bf49c38c6122163bc29684: Generated new TSK 0
I20260928 10:58:20.575899 9448 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 10:58:20.612169 9464 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 10:58:20.612277 9464 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:58:20.613540 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:20.592979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.3.252.126:41903)
0928 10:58:20.593163 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:20.593170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:20.593619 (+ 449us) server_negotiation.cc:407] Connection header received
0928 10:58:20.593869 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:20.593874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:20.593970 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:20.594136 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:58:20.594521 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:20.595648 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:20.597049 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:20.597723 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:20.597911 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:20.598367 (+ 456us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:58:20.612842 (+ 14475us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 10:58:20.613244 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50210: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 10:58:20.613574 9463 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:20.592930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:41903 (local address 127.0.0.1:50210)
0928 10:58:20.593530 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:20.593546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:20.593675 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:20.594140 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:20.594144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:20.594153 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:58:20.594376 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:20.594384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:20.595768 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:20.595772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:20.596906 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:20.596919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:20.597936 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:20.597942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:20.598081 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:20.598116 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:58:20.613077 (+ 14961us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 10:58:20.613273 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.252.126:41903: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":315,"threads_started":1}
W20260928 10:58:20.613660 9437 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50210: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 10:58:20.615010 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9368
I20260928 10:58:20.616075 9429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:20.706797 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9368
2026-09-28T10:58:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (694 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 10:58:20.758178 4081 mini_oidc.cc:152] Starting JWKS server
I20260928 10:58:20.759940 4081 webserver.cc:460] Webserver started at https://127.0.0.1:33585/ using document root <none> and password file <none>
I20260928 10:58:20.760128 4081 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:58:20.760679 4081 webserver.cc:460] Webserver started at http://127.0.0.1:36925/ using document root <none> and password file <none>
2026-09-28T10:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:20Z Disabled control of system clock
I20260928 10:58:20.784041 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:43089
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:43089
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33585/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 10:58:21.085152 9471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:21.085440 9471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:21.085536 9471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:21.092594 9471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:21.092700 9471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:21.092746 9471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:21.092801 9471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:21.092844 9471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:58:21.092900 9471 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:58:21.092937 9471 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33585/jwks.json
I20260928 10:58:21.101963 9471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:43089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:43089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33585/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:21.103192 9471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:21.104905 9471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:21.116106 9476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:21.116068 9477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:21.116135 9479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:21.116479 9471 server_base.cc:1034] running on GCE node
I20260928 10:58:21.117476 9471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:21.118836 9471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:21.120043 9471 hybrid_clock.cc:648] HybridClock initialized: now 1790593101119986 us; error 42 us; skew 500 ppm
I20260928 10:58:21.120383 9471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:21.122440 9471 webserver.cc:460] Webserver started at http://127.3.252.126:46345/ using document root <none> and password file <none>
I20260928 10:58:21.123080 9471 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:21.123174 9471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:21.123484 9471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:21.126240 9471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "0aaf7080636e47c98e41538b83df5fd9"
format_stamp: "Formatted at 2026-09-28 10:58:21 on dist-test-slave-djjw"
server_key: "b037939799bbf7e80eb976c7b322b42d"
server_key_iv: "7b3af1a88b553a4cadd5b9dd06a4a452"
server_key_version: "encryptionkey@0"
I20260928 10:58:21.127023 9471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "0aaf7080636e47c98e41538b83df5fd9"
format_stamp: "Formatted at 2026-09-28 10:58:21 on dist-test-slave-djjw"
server_key: "b037939799bbf7e80eb976c7b322b42d"
server_key_iv: "7b3af1a88b553a4cadd5b9dd06a4a452"
server_key_version: "encryptionkey@0"
I20260928 10:58:21.131814 9471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:58:21.135115 9485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:21.136397 9471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:58:21.136561 9471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "0aaf7080636e47c98e41538b83df5fd9"
format_stamp: "Formatted at 2026-09-28 10:58:21 on dist-test-slave-djjw"
server_key: "b037939799bbf7e80eb976c7b322b42d"
server_key_iv: "7b3af1a88b553a4cadd5b9dd06a4a452"
server_key_version: "encryptionkey@0"
I20260928 10:58:21.136749 9471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:21.156939 9471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:21.157873 9471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:21.158167 9471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:21.175455 9471 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:43089
I20260928 10:58:21.175485 9536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:43089 every 8 connection(s)
I20260928 10:58:21.176738 9471 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:21.178108 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9471
I20260928 10:58:21.178296 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:21.178874 4081 external_mini_cluster.cc:1428] Setting key 9a1db9bdb391ddc224935ced99089e07
I20260928 10:58:21.181814 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:21.191133 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:21.180488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.3.252.126:43089)
0928 10:58:21.181336 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:21.181346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:21.181386 (+ 40us) server_negotiation.cc:407] Connection header received
0928 10:58:21.182917 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:21.182938 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:21.183383 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:21.183725 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:21.184279 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.185566 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.186841 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.187473 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.187630 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:21.188795 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:21.188803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:21.188819 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:21.189380 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:21.189461 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:21.189481 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:21.189600 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:21.189823 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 10:58:21.190178 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":264,"threads_started":1}
I20260928 10:58:21.192116 9537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9: Bootstrap starting.
I20260928 10:58:21.194612 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:58:21.195284 9537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:21.196218 9537 log.cc:824] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:21.199101 9537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9: No bootstrap required, opened a new log
I20260928 10:58:21.206203 9537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaf7080636e47c98e41538b83df5fd9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43089 } }
I20260928 10:58:21.206646 9537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:21.206751 9537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaf7080636e47c98e41538b83df5fd9, State: Initialized, Role: FOLLOWER
I20260928 10:58:21.207297 9537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaf7080636e47c98e41538b83df5fd9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43089 } }
I20260928 10:58:21.207497 9537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:21.207615 9537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:21.207762 9537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:21.209786 9537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaf7080636e47c98e41538b83df5fd9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43089 } }
I20260928 10:58:21.210314 9537 leader_election.cc:302] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aaf7080636e47c98e41538b83df5fd9; no voters:
I20260928 10:58:21.210872 9537 leader_election.cc:288] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:21.211009 9542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:21.211673 9542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [term 1 LEADER]: Becoming Leader. State: Replica: 0aaf7080636e47c98e41538b83df5fd9, State: Running, Role: LEADER
I20260928 10:58:21.212186 9542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaf7080636e47c98e41538b83df5fd9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43089 } }
I20260928 10:58:21.212775 9537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:21.214699 9544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aaf7080636e47c98e41538b83df5fd9. Latest consensus state: current_term: 1 leader_uuid: "0aaf7080636e47c98e41538b83df5fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaf7080636e47c98e41538b83df5fd9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43089 } } }
I20260928 10:58:21.215055 9544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:21.214926 9543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aaf7080636e47c98e41538b83df5fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaf7080636e47c98e41538b83df5fd9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43089 } } }
I20260928 10:58:21.215215 9543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:21.217168 9547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:21.222198 9547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:21.230780 9547 catalog_manager.cc:1347] Generated new cluster ID: ced683704c3746f281cdc1a13a1791a0
I20260928 10:58:21.230891 9547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:21.260434 9547 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:21.261813 9547 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:21.269436 9547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aaf7080636e47c98e41538b83df5fd9: Generated new TSK 0
I20260928 10:58:21.270377 9547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:58:21.290026 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:21.282478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.3.252.126:43089)
0928 10:58:21.282626 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:21.282631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:21.283129 (+ 498us) server_negotiation.cc:407] Connection header received
0928 10:58:21.283372 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:21.283376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:21.283451 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:21.283583 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:58:21.284026 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.284743 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.286135 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.286581 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.286696 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:21.287200 (+ 504us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:58:21.289360 (+ 2160us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33585/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 10:58:21.289680 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33585/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 10:58:21.290160 9540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33585/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 10:58:21.292281 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:21.282494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:43089 (local address 127.0.0.1:35032)
0928 10:58:21.283021 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:21.283036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:21.283176 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:21.283617 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:21.283621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:21.283634 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:58:21.283880 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:21.283889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.284882 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.284887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:21.285993 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:21.286005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.286768 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.286775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:21.286908 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:21.286945 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:58:21.289573 (+ 2628us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33585/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 10:58:21.291821 (+ 2248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.252.126:43089: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33585/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1927,"thread_start_us":275,"threads_started":1}
I20260928 10:58:21.293541 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9471
I20260928 10:58:21.294660 9532 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 10:58:21.305130 9567 webserver.cc:533] Webserver: sslize failed:
E20260928 10:58:21.305267 9567 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:58:21.386665 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9471
2026-09-28T10:58:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (680 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T10:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:21Z Disabled control of system clock
I20260928 10:58:21.463868 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:45791
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:45791
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/testchainca.cert with env {}
W20260928 10:58:21.779862 9572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:21.780186 9572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:21.780290 9572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:21.787683 9572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:21.787801 9572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:21.787840 9572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:21.787916 9572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:21.787977 9572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:58:21.788061 9572 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:58:21.788115 9572 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 10:58:21.798291 9572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:45791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:21.799599 9572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:21.800853 9572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:21.811893 9577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:21.811926 9578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:21.811862 9580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:21.812625 9572 server_base.cc:1034] running on GCE node
I20260928 10:58:21.813349 9572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:21.814741 9572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:21.815933 9572 hybrid_clock.cc:648] HybridClock initialized: now 1790593101815890 us; error 32 us; skew 500 ppm
I20260928 10:58:21.816247 9572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:58:21.818295 9572 webserver.cc:460] Webserver started at http://127.3.252.126:44407/ using document root <none> and password file <none>
I20260928 10:58:21.818929 9572 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:21.819046 9572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:21.819330 9572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:21.821985 9572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "c60febb09f6c4324be58e4aa25cd5ba6"
format_stamp: "Formatted at 2026-09-28 10:58:21 on dist-test-slave-djjw"
server_key: "695bb16cde2dec26ad05edfa33be83b0"
server_key_iv: "a850e48e55d83f60ad8b226a1c87f508"
server_key_version: "encryptionkey@0"
I20260928 10:58:21.822722 9572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "c60febb09f6c4324be58e4aa25cd5ba6"
format_stamp: "Formatted at 2026-09-28 10:58:21 on dist-test-slave-djjw"
server_key: "695bb16cde2dec26ad05edfa33be83b0"
server_key_iv: "a850e48e55d83f60ad8b226a1c87f508"
server_key_version: "encryptionkey@0"
I20260928 10:58:21.827494 9572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:58:21.830608 9586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:21.831812 9572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 10:58:21.831985 9572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "c60febb09f6c4324be58e4aa25cd5ba6"
format_stamp: "Formatted at 2026-09-28 10:58:21 on dist-test-slave-djjw"
server_key: "695bb16cde2dec26ad05edfa33be83b0"
server_key_iv: "a850e48e55d83f60ad8b226a1c87f508"
server_key_version: "encryptionkey@0"
I20260928 10:58:21.832209 9572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:21.858991 9572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:21.859771 9572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:21.860003 9572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:21.876411 9572 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:45791
I20260928 10:58:21.876418 9637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:45791 every 8 connection(s)
I20260928 10:58:21.877686 9572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:21.881417 9638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:21.882323 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9572
I20260928 10:58:21.882530 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:21.883335 4081 external_mini_cluster.cc:1428] Setting key 43719b46f407c60c872fc7d01994a99a
I20260928 10:58:21.893724 9638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6: Bootstrap starting.
I20260928 10:58:21.895780 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:21.885303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.3.252.126:45791)
0928 10:58:21.886243 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:21.886256 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:21.886299 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:58:21.887885 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:21.887907 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:21.888363 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:21.888709 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:21.889219 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.890499 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.891722 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.892367 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.892529 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:21.893624 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:21.893632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:21.893644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:58:21.894198 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:21.894269 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:58:21.894288 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:21.894386 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:21.894605 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 10:58:21.894881 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":304,"threads_started":1}
I20260928 10:58:21.896991 9638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:21.898062 9638 log.cc:824] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:21.899581 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:58:21.901247 9638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6: No bootstrap required, opened a new log
I20260928 10:58:21.908607 9638 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45791 } }
I20260928 10:58:21.908934 9638 raft_consensus.cc:374] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:21.909004 9638 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60febb09f6c4324be58e4aa25cd5ba6, State: Initialized, Role: FOLLOWER
I20260928 10:58:21.909461 9638 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45791 } }
I20260928 10:58:21.909636 9638 raft_consensus.cc:388] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:21.909729 9638 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:21.909883 9638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:21.912041 9638 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45791 } }
I20260928 10:58:21.912474 9638 leader_election.cc:302] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c60febb09f6c4324be58e4aa25cd5ba6; no voters:
I20260928 10:58:21.912986 9638 leader_election.cc:288] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:21.913126 9643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:21.913991 9643 raft_consensus.cc:686] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [term 1 LEADER]: Becoming Leader. State: Replica: c60febb09f6c4324be58e4aa25cd5ba6, State: Running, Role: LEADER
I20260928 10:58:21.914523 9643 consensus_queue.cc:237] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45791 } }
I20260928 10:58:21.914880 9638 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:21.916873 9644 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45791 } } }
I20260928 10:58:21.917146 9644 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:21.918605 9648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:21.918550 9645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60febb09f6c4324be58e4aa25cd5ba6. Latest consensus state: current_term: 1 leader_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60febb09f6c4324be58e4aa25cd5ba6" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45791 } } }
I20260928 10:58:21.918877 9645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:21.924145 9648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:21.932164 9648 catalog_manager.cc:1347] Generated new cluster ID: dae8c1f0a68a47f4af941299a2cd048b
I20260928 10:58:21.932272 9648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:21.946870 9648 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:21.948096 9648 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:21.953469 9648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c60febb09f6c4324be58e4aa25cd5ba6: Generated new TSK 0
I20260928 10:58:21.954303 9648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:58:22.025046 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:21.967033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.3.252.126:45791)
0928 10:58:21.967232 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:21.967240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:21.967642 (+ 402us) server_negotiation.cc:407] Connection header received
0928 10:58:21.967840 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:21.967843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:21.967906 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:21.968052 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:58:21.968539 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.969241 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.971278 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.971710 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.971808 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:21.972356 (+ 548us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:58:22.024487 (+ 52131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 10:58:22.024797 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33196: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 10:58:22.025177 9641 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33196: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 10:58:22.025511 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:21.966974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:45791 (local address 127.0.0.1:33196)
0928 10:58:21.967516 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:21.967536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:21.967692 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:21.968083 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:21.968088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:21.968101 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:58:21.968403 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:21.968415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.969379 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.969385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:21.971132 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:21.971148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:21.971873 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:21.971879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:21.972071 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:21.972129 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:58:22.024794 (+ 52665us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 10:58:22.025112 (+ 318us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.252.126:45791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":264,"threads_started":1}
I20260928 10:58:22.027498 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9572
I20260928 10:58:22.028601 9633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:22.127907 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9572
2026-09-28T10:58:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (720 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (391 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (390 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:22 dist-test-slave-djjw krb5kdc[9682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:22 dist-test-slave-djjw krb5kdc[9682](info): set up 2 sockets
Sep 28 10:58:22 dist-test-slave-djjw krb5kdc[9682](info): commencing operation
krb5kdc: starting...
W20260928 10:58:23.914467 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:24 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:24.104978 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33699
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:24.415585 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:24.415884 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:24.416002 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:24.423450 9695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:24.423570 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:24.423616 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:24.423691 9695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:24.423744 9695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:24.433686 9695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:24.435024 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:24.436275 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:24.447521 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:24.447559 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:24.447494 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:24.448557 9695 server_base.cc:1034] running on GCE node
I20260928 10:58:24.449298 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:24.450757 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:24.451969 9695 hybrid_clock.cc:648] HybridClock initialized: now 1790593104451911 us; error 51 us; skew 500 ppm
I20260928 10:58:24.452306 9695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:24 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:24.455355 9695 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:24.456851 9695 webserver.cc:460] Webserver started at http://127.3.252.126:37943/ using document root <none> and password file <none>
I20260928 10:58:24.457522 9695 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:24.457629 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:24.457978 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:24.460876 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "98da51d6b3a74baea2a530854a412622"
format_stamp: "Formatted at 2026-09-28 10:58:24 on dist-test-slave-djjw"
server_key: "6c3b7a5996145d21a7f052ed30bf02a7"
server_key_iv: "8512c9d8886feabc79f5a7331b897ecf"
server_key_version: "encryptionkey@0"
I20260928 10:58:24.461608 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "98da51d6b3a74baea2a530854a412622"
format_stamp: "Formatted at 2026-09-28 10:58:24 on dist-test-slave-djjw"
server_key: "6c3b7a5996145d21a7f052ed30bf02a7"
server_key_iv: "8512c9d8886feabc79f5a7331b897ecf"
server_key_version: "encryptionkey@0"
I20260928 10:58:24.466575 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 10:58:24.469760 9710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:24.471027 9695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:58:24.471169 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "98da51d6b3a74baea2a530854a412622"
format_stamp: "Formatted at 2026-09-28 10:58:24 on dist-test-slave-djjw"
server_key: "6c3b7a5996145d21a7f052ed30bf02a7"
server_key_iv: "8512c9d8886feabc79f5a7331b897ecf"
server_key_version: "encryptionkey@0"
I20260928 10:58:24.471339 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:24.508450 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:24.512681 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:24.512959 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:24.529627 9695 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:33699
I20260928 10:58:24.529666 9761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:33699 every 8 connection(s)
I20260928 10:58:24.530958 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:24.533238 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9695
I20260928 10:58:24.533492 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:24.534336 4081 external_mini_cluster.cc:1428] Setting key 46115073bc3e770b8dda78c71a95288d
I20260928 10:58:24.535305 9762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:58:24 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:24.549500 9762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622: Bootstrap starting.
I20260928 10:58:24.552675 9762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:24.553871 9762 log.cc:824] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:24.557070 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:24.536349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.3.252.126:33699)
0928 10:58:24.537081 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:24.537092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:24.537134 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:58:24.538674 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:24.538696 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:24.539087 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:24.539426 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:24.541095 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:24.542552 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:24.544272 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:24.544951 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:24.545112 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:24.549725 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:24.549733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:24.549749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:24.549788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:24.552637 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:24.553442 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:24.553450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:24.553456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:24.553628 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:24.554063 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:24.554068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:24.554072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:24.554609 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:24.554829 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:24.555396 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 10:58:24.555817 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
I20260928 10:58:24.557176 9762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622: No bootstrap required, opened a new log
I20260928 10:58:24.564548 9762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98da51d6b3a74baea2a530854a412622" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33699 } }
I20260928 10:58:24.564870 9762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:24.564939 9762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98da51d6b3a74baea2a530854a412622, State: Initialized, Role: FOLLOWER
I20260928 10:58:24.565476 9762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98da51d6b3a74baea2a530854a412622" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33699 } }
I20260928 10:58:24.565733 9762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:24.565876 9762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:24.566042 9762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:24.568064 9762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98da51d6b3a74baea2a530854a412622" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33699 } }
I20260928 10:58:24.568478 9762 leader_election.cc:302] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98da51d6b3a74baea2a530854a412622; no voters:
I20260928 10:58:24.568957 9762 leader_election.cc:288] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:24.569088 9767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:24.569696 9767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [term 1 LEADER]: Becoming Leader. State: Replica: 98da51d6b3a74baea2a530854a412622, State: Running, Role: LEADER
I20260928 10:58:24.570238 9767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98da51d6b3a74baea2a530854a412622" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33699 } }
I20260928 10:58:24.570895 9762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:24.572798 9769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98da51d6b3a74baea2a530854a412622. Latest consensus state: current_term: 1 leader_uuid: "98da51d6b3a74baea2a530854a412622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98da51d6b3a74baea2a530854a412622" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33699 } } }
I20260928 10:58:24.573173 9769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:24.574067 9768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98da51d6b3a74baea2a530854a412622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98da51d6b3a74baea2a530854a412622" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33699 } } }
I20260928 10:58:24.574357 9768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:24.576375 9774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:24.581624 9774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:24.590451 9774 catalog_manager.cc:1347] Generated new cluster ID: bb0511f693d341d0b3872483ce3a7e73
I20260928 10:58:24.590575 9774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:24.620409 9774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:24.621732 9774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:24.632187 9774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98da51d6b3a74baea2a530854a412622: Generated new TSK 0
I20260928 10:58:24.633045 9774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:58:24.685676 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33699
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:25.001257 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:25.001576 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:25.001708 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:25.008939 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:25.009060 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:25.009217 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:25.018958 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:25.020208 9788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:25.021778 9788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:25.034123 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:25.034030 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:25.034090 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:25.034981 9788 server_base.cc:1034] running on GCE node
I20260928 10:58:25.035614 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:25.036567 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:25.037799 9788 hybrid_clock.cc:648] HybridClock initialized: now 1790593105037747 us; error 38 us; skew 500 ppm
I20260928 10:58:25.038204 9788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:25 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:25.044526 9788 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:58:25.046180 9788 webserver.cc:460] Webserver started at http://127.3.252.65:36243/ using document root <none> and password file <none>
I20260928 10:58:25.046850 9788 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:25.046967 9788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:25.047256 9788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:25.050246 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "e52b7b6e8750458ab21536dabcc4d857"
format_stamp: "Formatted at 2026-09-28 10:58:25 on dist-test-slave-djjw"
server_key: "721ac215403343ab79eeeadb453b467f"
server_key_iv: "fdd79700cd6e0489c40d75e0188c68c1"
server_key_version: "encryptionkey@0"
I20260928 10:58:25.051011 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "e52b7b6e8750458ab21536dabcc4d857"
format_stamp: "Formatted at 2026-09-28 10:58:25 on dist-test-slave-djjw"
server_key: "721ac215403343ab79eeeadb453b467f"
server_key_iv: "fdd79700cd6e0489c40d75e0188c68c1"
server_key_version: "encryptionkey@0"
I20260928 10:58:25.056232 9788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:25.059605 9803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:25.060926 9788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:58:25.061082 9788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "e52b7b6e8750458ab21536dabcc4d857"
format_stamp: "Formatted at 2026-09-28 10:58:25 on dist-test-slave-djjw"
server_key: "721ac215403343ab79eeeadb453b467f"
server_key_iv: "fdd79700cd6e0489c40d75e0188c68c1"
server_key_version: "encryptionkey@0"
I20260928 10:58:25.061273 9788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:25.094318 9788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:25.098326 9788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:25.098703 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:25.099713 9788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:25.101018 9788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:25.101097 9788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:25.101217 9788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:25.101267 9788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:25.131018 9788 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:40007
I20260928 10:58:25.131103 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:40007 every 8 connection(s)
I20260928 10:58:25.132373 9788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:25.136804 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9788
I20260928 10:58:25.136978 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:25.137550 4081 external_mini_cluster.cc:1428] Setting key 5830e83f6a19698153c4c0f16f116c55
Sep 28 10:58:25 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:25.153476 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:25.134770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:44661 (local address 127.3.252.126:33699)
0928 10:58:25.135167 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:25.135172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:25.136109 (+ 937us) server_negotiation.cc:407] Connection header received
0928 10:58:25.137865 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:25.137871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:25.137950 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:25.138046 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:25.140240 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.141078 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.143118 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.143506 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.143598 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:25.147468 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:25.147474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:25.147478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:25.147517 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:25.150524 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:25.151231 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:25.151237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:25.151241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:25.151372 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:25.151760 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:25.151764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:25.151767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:25.152088 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:25.152226 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:25.152939 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 10:58:25.153143 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":194,"threads_started":1}
I20260928 10:58:25.154455 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:25.135291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33699 (local address 127.3.252.65:44661)
0928 10:58:25.135947 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:25.135983 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:25.137591 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:25.138323 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:25.138331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:25.139013 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:25.139987 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:25.140013 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.141235 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.141240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:25.142969 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:25.142982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.143712 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.143719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:25.143970 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:25.144676 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:25.144703 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:25.147251 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:25.150690 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:25.150697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:25.150711 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:25.151127 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:25.151479 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:25.151482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:25.151485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:25.151665 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:25.152223 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:25.152229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:25.152572 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 10:58:25.152821 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 10:58:25.153245 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":272,"threads_started":1}
I20260928 10:58:25.156059 9916 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33699
I20260928 10:58:25.156338 9916 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:25.157073 9916 heartbeater.cc:511] Master 127.3.252.126:33699 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 10:58:25.159132 9727 ts_manager.cc:194] Registered new tserver with Master: e52b7b6e8750458ab21536dabcc4d857 (127.3.252.65:40007)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
I20260928 10:58:25.160598 9727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:44661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:58:25.184679 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33699
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:25.504113 9922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:25.504396 9922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:25.504516 9922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:25.511513 9922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:25.511634 9922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:25.511775 9922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:25.521195 9922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:25.522490 9922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:25.523774 9922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:25.536173 9928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:25.536144 9927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:25.536707 9930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:25.536996 9922 server_base.cc:1034] running on GCE node
I20260928 10:58:25.537590 9922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:25.538532 9922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:25.539721 9922 hybrid_clock.cc:648] HybridClock initialized: now 1790593105539668 us; error 47 us; skew 500 ppm
I20260928 10:58:25.540014 9922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:25 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:25.543316 9922 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:58:25.546351 9922 webserver.cc:460] Webserver started at http://127.3.252.66:34247/ using document root <none> and password file <none>
I20260928 10:58:25.547000 9922 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:25.547111 9922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:25.547379 9922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:25.550151 9922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "9d47d812881e4168a3ce8fa005925564"
format_stamp: "Formatted at 2026-09-28 10:58:25 on dist-test-slave-djjw"
server_key: "7089563ec24c5685001f4be20e373f74"
server_key_iv: "d7409af15fd61494f58e380b4f4484cb"
server_key_version: "encryptionkey@0"
I20260928 10:58:25.550871 9922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "9d47d812881e4168a3ce8fa005925564"
format_stamp: "Formatted at 2026-09-28 10:58:25 on dist-test-slave-djjw"
server_key: "7089563ec24c5685001f4be20e373f74"
server_key_iv: "d7409af15fd61494f58e380b4f4484cb"
server_key_version: "encryptionkey@0"
I20260928 10:58:25.555698 9922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:25.559206 9937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:25.560483 9922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:58:25.560627 9922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "9d47d812881e4168a3ce8fa005925564"
format_stamp: "Formatted at 2026-09-28 10:58:25 on dist-test-slave-djjw"
server_key: "7089563ec24c5685001f4be20e373f74"
server_key_iv: "d7409af15fd61494f58e380b4f4484cb"
server_key_version: "encryptionkey@0"
I20260928 10:58:25.560798 9922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:25.582880 9922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:25.586973 9922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:25.587225 9922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:25.588124 9922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:25.589428 9922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:25.589507 9922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:25.589619 9922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:25.589670 9922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:25.619413 9922 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:43569
I20260928 10:58:25.619460 10049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:43569 every 8 connection(s)
I20260928 10:58:25.620613 9922 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:25.623780 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 9922
I20260928 10:58:25.623930 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:25.624369 4081 external_mini_cluster.cc:1428] Setting key 5aa37c14e8667caf2a3561c8241d155e
Sep 28 10:58:25 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:25.639189 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:25.623035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:49717 (local address 127.3.252.126:33699)
0928 10:58:25.623195 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:25.623199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:25.624370 (+ 1171us) server_negotiation.cc:407] Connection header received
0928 10:58:25.626321 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:25.626326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:25.626392 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:25.626473 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:25.627858 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.628550 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.630031 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.630454 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.630564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:25.634002 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:25.634007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:25.634010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:25.634041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:25.636262 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:25.636932 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:25.636936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:25.636939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:25.637061 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:25.637455 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:25.637459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:25.637461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:25.637722 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:25.637874 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:25.638633 (+ 759us) server_negotiation.cc:299] Negotiation successful
0928 10:58:25.638851 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:58:25.640134 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:25.623435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33699 (local address 127.3.252.66:49717)
0928 10:58:25.624196 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:25.624246 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:25.626083 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:25.626651 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:25.626659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:25.627164 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:25.627697 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:25.627718 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.628682 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.628686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:25.629901 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:25.629914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:25.630660 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:25.630667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:25.630880 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:25.631487 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:25.631511 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:25.633760 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:25.636391 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:25.636400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:25.636417 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:25.636813 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:25.637163 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:25.637168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:25.637172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:25.637347 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:25.637861 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:25.637869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:25.638268 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 10:58:25.638499 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 10:58:25.639013 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 10:58:25.641717 10050 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33699
I20260928 10:58:25.642019 10050 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:25.642735 10050 heartbeater.cc:511] Master 127.3.252.126:33699 requested a full tablet report, sending...
I20260928 10:58:25.644212 9727 ts_manager.cc:194] Registered new tserver with Master: 9d47d812881e4168a3ce8fa005925564 (127.3.252.66:43569)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
I20260928 10:58:25.645102 9727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:49717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:58:25.675169 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33699
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:25.984547 10055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:25.984826 10055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:25.984916 10055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:25.991959 10055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:25.992060 10055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:25.992174 10055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:26.002041 10055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:26.003244 10055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:26.004468 10055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:26.016585 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:26.016893 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:26.016952 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:26.017321 10055 server_base.cc:1034] running on GCE node
I20260928 10:58:26.017951 10055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:26.018781 10055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:26.019986 10055 hybrid_clock.cc:648] HybridClock initialized: now 1790593106019933 us; error 40 us; skew 500 ppm
I20260928 10:58:26.020291 10055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.023208 10055 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:58:26.024539 10055 webserver.cc:460] Webserver started at http://127.3.252.67:32881/ using document root <none> and password file <none>
I20260928 10:58:26.025169 10055 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:26.025280 10055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:26.025560 10055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:26.028352 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "a69aeaa0ea054274a84f163e6a4f40af"
format_stamp: "Formatted at 2026-09-28 10:58:26 on dist-test-slave-djjw"
server_key: "d21be1a9893153d9034c82b6a78deb00"
server_key_iv: "a9e1642ac141768d4e9fb7c73283693e"
server_key_version: "encryptionkey@0"
I20260928 10:58:26.029078 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "a69aeaa0ea054274a84f163e6a4f40af"
format_stamp: "Formatted at 2026-09-28 10:58:26 on dist-test-slave-djjw"
server_key: "d21be1a9893153d9034c82b6a78deb00"
server_key_iv: "a9e1642ac141768d4e9fb7c73283693e"
server_key_version: "encryptionkey@0"
I20260928 10:58:26.034010 10055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 10:58:26.037314 10070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:26.038638 10055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:26.038781 10055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "a69aeaa0ea054274a84f163e6a4f40af"
format_stamp: "Formatted at 2026-09-28 10:58:26 on dist-test-slave-djjw"
server_key: "d21be1a9893153d9034c82b6a78deb00"
server_key_iv: "a9e1642ac141768d4e9fb7c73283693e"
server_key_version: "encryptionkey@0"
I20260928 10:58:26.038959 10055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:26.062335 10055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:26.066777 10055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:26.067037 10055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:26.067950 10055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:26.069214 10055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:26.069289 10055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:26.069401 10055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:26.069451 10055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:26.098675 10055 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:34411
I20260928 10:58:26.098726 10182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:34411 every 8 connection(s)
I20260928 10:58:26.099907 10055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:26.105702 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 10055
I20260928 10:58:26.105901 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:26.106421 4081 external_mini_cluster.cc:1428] Setting key f831cb83a31b79f32966a89c8da7c12a
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:26.118618 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.102283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:43301 (local address 127.3.252.126:33699)
0928 10:58:26.102498 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.102504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.103628 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 10:58:26.105305 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.105309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.105376 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.105459 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.107314 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.108220 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.109882 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.110287 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.110377 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.113412 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:26.113416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:26.113419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:26.113450 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:26.115957 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.116691 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.116695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.116697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.116813 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.117148 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.117150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.117152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.117390 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:26.117491 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:26.118226 (+ 735us) server_negotiation.cc:299] Negotiation successful
0928 10:58:26.118380 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:58:26.119793 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.102775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33699 (local address 127.3.252.67:43301)
0928 10:58:26.103468 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:26.103509 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:26.105054 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:26.105725 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:26.105736 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:26.106436 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:26.107141 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:26.107165 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.108354 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.108359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:26.109741 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:26.109752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.110452 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.110457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:26.110650 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.111244 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:26.111265 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:26.113236 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:26.116103 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:26.116109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:26.116122 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:26.116581 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:26.116892 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:26.116895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:26.116897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:26.117068 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:26.117525 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:26.117532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:26.117885 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 10:58:26.118135 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 10:58:26.118508 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":267,"threads_started":1}
I20260928 10:58:26.121251 10183 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33699
I20260928 10:58:26.121500 10183 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:26.122382 10183 heartbeater.cc:511] Master 127.3.252.126:33699 requested a full tablet report, sending...
I20260928 10:58:26.123920 9727 ts_manager.cc:194] Registered new tserver with Master: a69aeaa0ea054274a84f163e6a4f40af (127.3.252.67:34411)
I20260928 10:58:26.124991 9727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:43301
I20260928 10:58:26.134351 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/test-admin.keytab.
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.159299 4081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260928 10:58:26.164021 9916 heartbeater.cc:503] Master 127.3.252.126:33699 was elected leader, sending a full tablet report...
W20260928 10:58:26.164299 10190 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:26.161905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.3.252.65:40007)
0928 10:58:26.162500 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.162508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.162533 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:58:26.162602 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.162609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.162803 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.162954 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.163794 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47848: BlockingRecv error: recv got EOF from 127.0.0.1:47848 (error 108)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":283,"threads_started":1}
W20260928 10:58:26.164474 4081 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc)
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.166203 4081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.167469 4081 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:26.182070 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.168954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.3.252.65:40007)
0928 10:58:26.169136 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.169142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.169272 (+ 130us) server_negotiation.cc:407] Connection header received
0928 10:58:26.169395 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.169399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.169455 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.169531 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.170597 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.171565 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.172761 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.173286 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.173377 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.176838 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:26.176850 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:26.176865 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:26.176895 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:26.179453 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.180048 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.180055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.180057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.180174 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.180529 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.180532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.180534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.180875 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:26.181145 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:26.181494 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 10:58:26.181752 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 10:58:26.187450 10190 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:26.185777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.3.252.65:40007)
0928 10:58:26.185972 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.185978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.186234 (+ 256us) server_negotiation.cc:407] Connection header received
0928 10:58:26.186374 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.186379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.186432 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.186508 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.187234 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47872: BlockingRecv error: recv got EOF from 127.0.0.1:47872 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 10:58:26.187738 4081 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc' of type FILE
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.189082 4081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.190385 4081 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:26.202735 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.191556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.3.252.65:40007)
0928 10:58:26.191736 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.191742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.191886 (+ 144us) server_negotiation.cc:407] Connection header received
0928 10:58:26.191989 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.191992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.192043 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.192114 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.192930 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.193621 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.194761 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.195157 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.195244 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.198328 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:26.198333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:26.198336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:26.198379 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:26.200720 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.201226 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.201230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.201232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.201336 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.201596 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.201599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.201601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.201880 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:26.202076 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:26.202283 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 10:58:26.202427 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 10:58:26.206452 10190 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:26.205028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.3.252.65:40007)
0928 10:58:26.205161 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.205165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.205441 (+ 276us) server_negotiation.cc:407] Connection header received
0928 10:58:26.205555 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.205559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.205623 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.205696 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.206285 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47898: BlockingRecv error: recv got EOF from 127.0.0.1:47898 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 10:58:26.206678 4081 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790593056029081-4081-0/krb5kdc/krb5cc)
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.208127 4081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:26.209457 4081 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:26.221889 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.210688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.3.252.65:40007)
0928 10:58:26.210871 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.210877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.211185 (+ 308us) server_negotiation.cc:407] Connection header received
0928 10:58:26.211325 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.211329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.211381 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.211455 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.212331 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.213006 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.214172 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.214560 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.214650 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.217702 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:26.217707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:26.217710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:26.217739 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:26.219923 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.220444 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.220447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.220450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.220580 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.220835 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.220837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.220839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.221059 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:26.221286 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:26.221401 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 10:58:26.221544 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:58:26.223426 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9788
I20260928 10:58:26.234450 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.224047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.3.252.65:40007)
0928 10:58:26.224197 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.224203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.224486 (+ 283us) server_negotiation.cc:407] Connection header received
0928 10:58:26.224625 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.224630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.224683 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.224758 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.225613 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.226337 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.227454 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.227870 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.227972 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.229816 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:26.229821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:26.229824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:26.229882 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:26.232110 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.232682 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.232688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.232692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.232812 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.233138 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.233143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.233147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.233394 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:26.233628 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:26.233967 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 10:58:26.234159 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:58:26.234992 9911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:26.316534 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9788
I20260928 10:58:26.330111 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9922
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:58:26.344897 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.331119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.3.252.66:43569)
0928 10:58:26.331645 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.331653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.331678 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:58:26.331733 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.331741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.331956 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.332084 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.332997 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.334036 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.335273 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.335826 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.335919 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.339491 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:26.339502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:26.339516 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:26.339555 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:26.342368 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.342981 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.342987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.342990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.343111 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.343552 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.343556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.343558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.343897 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:26.344117 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:26.344354 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 10:58:26.344544 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
I20260928 10:58:26.346102 10045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:26.426538 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9922
I20260928 10:58:26.439141 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 10055
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[9682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:58:26.454501 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:26.440065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.3.252.67:34411)
0928 10:58:26.440673 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:26.440679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:26.440706 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:58:26.440761 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:26.440767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:26.440971 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:26.441131 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:26.442186 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.443224 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.444446 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:26.444991 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:26.445105 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:26.448938 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:26.448950 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:26.448965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:26.449000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:26.451642 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.452349 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.452357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.452360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.452477 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:26.452803 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:26.452807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:26.452809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:26.453171 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:26.453395 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:26.453779 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 10:58:26.454088 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":284,"threads_started":1}
I20260928 10:58:26.455845 10178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:26.536392 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 10055
I20260928 10:58:26.548683 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 9695
I20260928 10:58:26.549769 9757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:26.649137 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 9695
2026-09-28T10:58:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3744 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[10210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[10210](info): set up 2 sockets
Sep 28 10:58:26 dist-test-slave-djjw krb5kdc[10210](info): commencing operation
krb5kdc: starting...
W20260928 10:58:27.658370 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:27 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab.
Entry for principal oryx/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab.
I20260928 10:58:27.854528 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33045
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab
--principal=oryx/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:33045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:28.161739 10223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:28.162070 10223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:28.162209 10223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:28.169317 10223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:28.169420 10223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:28.169472 10223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:28.169548 10223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:28.169603 10223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:28.179354 10223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:33045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab
--principal=oryx/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:28.180646 10223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:28.181944 10223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:28.193056 10228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:28.193011 10229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:28.193192 10231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:28.193776 10223 server_base.cc:1034] running on GCE node
I20260928 10:58:28.194628 10223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:28.196023 10223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:28.197220 10223 hybrid_clock.cc:648] HybridClock initialized: now 1790593108197179 us; error 33 us; skew 500 ppm
I20260928 10:58:28.197551 10223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:28 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593108, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:28.200429 10223 init.cc:377] Logged in from keytab as oryx/127.3.252.126@KRBTEST.COM (short username oryx)
I20260928 10:58:28.201864 10223 webserver.cc:460] Webserver started at http://127.3.252.126:33073/ using document root <none> and password file <none>
I20260928 10:58:28.202522 10223 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:28.202633 10223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:28.202908 10223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:28.205873 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "7f1cb973abd94f638b2275484463e071"
format_stamp: "Formatted at 2026-09-28 10:58:28 on dist-test-slave-djjw"
server_key: "7b4c2576a50195e25fc7deb0e8fecedf"
server_key_iv: "7a635e716b52a5760beb9109ccea3378"
server_key_version: "encryptionkey@0"
I20260928 10:58:28.206629 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "7f1cb973abd94f638b2275484463e071"
format_stamp: "Formatted at 2026-09-28 10:58:28 on dist-test-slave-djjw"
server_key: "7b4c2576a50195e25fc7deb0e8fecedf"
server_key_iv: "7a635e716b52a5760beb9109ccea3378"
server_key_version: "encryptionkey@0"
I20260928 10:58:28.211835 10223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 10:58:28.215390 10238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:28.216774 10223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:58:28.216935 10223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "7f1cb973abd94f638b2275484463e071"
format_stamp: "Formatted at 2026-09-28 10:58:28 on dist-test-slave-djjw"
server_key: "7b4c2576a50195e25fc7deb0e8fecedf"
server_key_iv: "7a635e716b52a5760beb9109ccea3378"
server_key_version: "encryptionkey@0"
I20260928 10:58:28.217119 10223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:28.228219 10223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:28.232098 10223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:28.232350 10223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:28.249765 10223 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:33045
I20260928 10:58:28.249861 10289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:33045 every 8 connection(s)
I20260928 10:58:28.251117 10223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:28.252156 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 10223
I20260928 10:58:28.252377 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:28.253193 4081 external_mini_cluster.cc:1428] Setting key 51660f5c8f2bbfc875edf49ac2d4e4f5
I20260928 10:58:28.255666 10290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:58:28 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:28.267973 10290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: Bootstrap starting.
I20260928 10:58:28.271888 10290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:28.273005 10290 log.cc:824] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:28.276242 10290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: No bootstrap required, opened a new log
I20260928 10:58:28.276353 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:28.255186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.3.252.126:33045)
0928 10:58:28.255949 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:28.255960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:28.255999 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:58:28.257509 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:28.257528 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:28.257946 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:28.258269 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:28.259988 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:28.261391 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:28.263086 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:28.263824 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:28.263965 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:28.268340 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:28.268351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:28.268371 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:28.268423 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:28.272106 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:28.272833 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:28.272842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:28.272850 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:28.273045 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:28.273394 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:28.273399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:28.273402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:28.274036 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:28.274273 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:28.274706 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 10:58:28.275104 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":560,"thread_start_us":360,"threads_started":1}
I20260928 10:58:28.283521 10290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } }
I20260928 10:58:28.283866 10290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:28.283936 10290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1cb973abd94f638b2275484463e071, State: Initialized, Role: FOLLOWER
I20260928 10:58:28.284428 10290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } }
I20260928 10:58:28.284627 10290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:28.284719 10290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:28.284842 10290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:28.286968 10290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } }
I20260928 10:58:28.287436 10290 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1cb973abd94f638b2275484463e071; no voters:
I20260928 10:58:28.288064 10290 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:28.288195 10295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:28.288880 10295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 1 LEADER]: Becoming Leader. State: Replica: 7f1cb973abd94f638b2275484463e071, State: Running, Role: LEADER
I20260928 10:58:28.289402 10295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } }
I20260928 10:58:28.290143 10290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:28.292196 10296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } }
I20260928 10:58:28.292584 10296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:28.293282 10297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1cb973abd94f638b2275484463e071. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } }
I20260928 10:58:28.293576 10297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:28.294939 10304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:28.300000 10304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:28.309116 10304 catalog_manager.cc:1347] Generated new cluster ID: b7a8d97379564ff68befb672b741c40a
I20260928 10:58:28.309235 10304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:28.327761 10304 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:28.329078 10304 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:28.345206 10304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: Generated new TSK 0
I20260928 10:58:28.346182 10304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab.
Entry for principal oryx/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab.
I20260928 10:58:28.401939 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33045
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab
--principal=oryx/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:28.722319 10316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:28.722627 10316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:28.722721 10316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:28.730110 10316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:28.730221 10316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:28.730345 10316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:28.740103 10316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab
--principal=oryx/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:28.741312 10316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:28.742583 10316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:28.754825 10321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:28.754938 10324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:28.754959 10322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:28.755998 10316 server_base.cc:1034] running on GCE node
I20260928 10:58:28.756589 10316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:28.758066 10316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:28.759296 10316 hybrid_clock.cc:648] HybridClock initialized: now 1790593108759222 us; error 66 us; skew 500 ppm
I20260928 10:58:28.759601 10316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:28 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593108, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:28.762544 10316 init.cc:377] Logged in from keytab as oryx/127.3.252.65@KRBTEST.COM (short username oryx)
I20260928 10:58:28.764026 10316 webserver.cc:460] Webserver started at http://127.3.252.65:45997/ using document root <none> and password file <none>
I20260928 10:58:28.764683 10316 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:28.764793 10316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:28.765060 10316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:28.767890 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "75048a2576194db5ba5cfcb17b09babc"
format_stamp: "Formatted at 2026-09-28 10:58:28 on dist-test-slave-djjw"
server_key: "1f7d38f3eb274e571a0c3fac941a046b"
server_key_iv: "b7558ef8c72fa1669472848fba175556"
server_key_version: "encryptionkey@0"
I20260928 10:58:28.768623 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "75048a2576194db5ba5cfcb17b09babc"
format_stamp: "Formatted at 2026-09-28 10:58:28 on dist-test-slave-djjw"
server_key: "1f7d38f3eb274e571a0c3fac941a046b"
server_key_iv: "b7558ef8c72fa1669472848fba175556"
server_key_version: "encryptionkey@0"
I20260928 10:58:28.773751 10316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:58:28.777379 10331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:28.778867 10316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 10:58:28.779021 10316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "75048a2576194db5ba5cfcb17b09babc"
format_stamp: "Formatted at 2026-09-28 10:58:28 on dist-test-slave-djjw"
server_key: "1f7d38f3eb274e571a0c3fac941a046b"
server_key_iv: "b7558ef8c72fa1669472848fba175556"
server_key_version: "encryptionkey@0"
I20260928 10:58:28.779201 10316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:28.795257 10316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:28.799335 10316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:28.799597 10316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:28.800503 10316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:28.801801 10316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:28.801908 10316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:28.802026 10316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:28.802083 10316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:28.831626 10316 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:42053
I20260928 10:58:28.831674 10443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:42053 every 8 connection(s)
I20260928 10:58:28.832839 10316 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:28.841058 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 10316
I20260928 10:58:28.841267 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:28.842067 4081 external_mini_cluster.cc:1428] Setting key 355712d9c10d647d30261586be302e41
Sep 28 10:58:28 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593108, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:28.853055 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:28.835290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:34927 (local address 127.3.252.126:33045)
0928 10:58:28.835734 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:28.835738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:28.836663 (+ 925us) server_negotiation.cc:407] Connection header received
0928 10:58:28.838441 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:28.838446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:28.838514 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:28.838608 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:28.840001 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:28.840724 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:28.842890 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:28.843379 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:28.843517 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:28.847131 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:28.847137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:28.847140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:28.847172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:28.849912 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:28.850807 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:28.850811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:28.850813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:28.850926 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:28.851345 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:28.851350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:28.851354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:28.851691 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:28.851835 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:28.852499 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 10:58:28.852718 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260928 10:58:28.853987 10447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:28.835794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.65:34927)
0928 10:58:28.836483 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:28.836530 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:28.838193 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:28.838810 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:28.838817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:28.839317 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:28.839847 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:28.839862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:28.840873 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:28.840879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:28.842753 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:28.842764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:28.843600 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:28.843605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:28.843832 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:28.844561 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:28.844584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:28.846891 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:28.850068 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:28.850078 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:28.850110 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:28.850680 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:28.851034 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:28.851039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:28.851042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:28.851239 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:28.851796 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:28.851802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:28.852127 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 10:58:28.852384 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 10:58:28.852782 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":258,"threads_started":1}
I20260928 10:58:28.855453 10444 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33045
I20260928 10:58:28.855772 10444 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:28.856575 10444 heartbeater.cc:511] Master 127.3.252.126:33045 requested a full tablet report, sending...
I20260928 10:58:28.858719 10255 ts_manager.cc:194] Registered new tserver with Master: 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:28.860100 10255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:34927
WARNING: no policy specified for oryx/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab.
Entry for principal oryx/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab.
I20260928 10:58:28.889084 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33045
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab
--principal=oryx/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:29.204185 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:29.204474 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:29.204608 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:29.211777 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:29.211891 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:29.212044 10450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:29.221917 10450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab
--principal=oryx/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:29.223214 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:29.224572 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:29.237104 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:29.237174 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:29.237202 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:29.238061 10450 server_base.cc:1034] running on GCE node
I20260928 10:58:29.238633 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:29.239526 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:29.240720 10450 hybrid_clock.cc:648] HybridClock initialized: now 1790593109240671 us; error 33 us; skew 500 ppm
I20260928 10:58:29.241003 10450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:29 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593109, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:29.244052 10450 init.cc:377] Logged in from keytab as oryx/127.3.252.66@KRBTEST.COM (short username oryx)
I20260928 10:58:29.245472 10450 webserver.cc:460] Webserver started at http://127.3.252.66:33567/ using document root <none> and password file <none>
I20260928 10:58:29.246217 10450 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:29.246330 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:29.246611 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:29.249511 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "739dff71c550428f8a56b3d96cdb08f0"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "5143b555f216f399c8208ee2cf984b71"
server_key_iv: "31743507a26c8fa79eeb3e37f0211ca4"
server_key_version: "encryptionkey@0"
I20260928 10:58:29.250255 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "739dff71c550428f8a56b3d96cdb08f0"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "5143b555f216f399c8208ee2cf984b71"
server_key_iv: "31743507a26c8fa79eeb3e37f0211ca4"
server_key_version: "encryptionkey@0"
I20260928 10:58:29.255431 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 10:58:29.258862 10465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:29.260288 10450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:58:29.260430 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "739dff71c550428f8a56b3d96cdb08f0"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "5143b555f216f399c8208ee2cf984b71"
server_key_iv: "31743507a26c8fa79eeb3e37f0211ca4"
server_key_version: "encryptionkey@0"
I20260928 10:58:29.260622 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:29.302340 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:29.306540 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:29.306823 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:29.307749 10450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:29.309103 10450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:29.309178 10450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:29.309293 10450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:29.309345 10450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:29.340328 10450 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:39347
I20260928 10:58:29.340376 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:39347 every 8 connection(s)
I20260928 10:58:29.341535 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:29.350659 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 10450
I20260928 10:58:29.350847 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:29.351400 4081 external_mini_cluster.cc:1428] Setting key 7b699f7fd83cd9b3e20aa4c8e5b2615b
Sep 28 10:58:29 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593109, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:29.361563 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:29.344053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:45861 (local address 127.3.252.126:33045)
0928 10:58:29.344281 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:29.344288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:29.345567 (+ 1279us) server_negotiation.cc:407] Connection header received
0928 10:58:29.347398 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:29.347404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:29.347499 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:29.347583 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:29.349048 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.349771 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.351874 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.352295 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.352410 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:29.355946 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:29.355951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:29.355955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:29.355997 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:29.358509 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:29.359369 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:29.359373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:29.359375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:29.359513 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:29.359910 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:29.359914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:29.359916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:29.360168 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:29.360313 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:29.360990 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 10:58:29.361231 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:58:29.362613 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:29.344604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:45861)
0928 10:58:29.345410 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:29.345453 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:29.347130 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:29.347825 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:29.347833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:29.348330 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:29.348881 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:29.348899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.349931 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.349935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:29.351705 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:29.351738 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.352498 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.352503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:29.352699 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:29.353323 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:29.353352 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:29.355694 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:29.358656 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:29.358665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:29.358683 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:29.359237 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:29.359616 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:29.359619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:29.359621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:29.359796 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:29.360293 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:29.360301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:29.360637 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 10:58:29.360869 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 10:58:29.361422 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":336,"threads_started":1}
I20260928 10:58:29.363950 10578 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33045
I20260928 10:58:29.364277 10578 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:29.365163 10578 heartbeater.cc:511] Master 127.3.252.126:33045 requested a full tablet report, sending...
I20260928 10:58:29.366663 10255 ts_manager.cc:194] Registered new tserver with Master: 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66:39347)
I20260928 10:58:29.367509 10255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:45861
WARNING: no policy specified for oryx/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab.
Entry for principal oryx/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab.
I20260928 10:58:29.397075 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33045
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab
--principal=oryx/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:29.707433 10583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:29.707710 10583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:29.707827 10583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:29.715098 10583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:29.715206 10583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:29.715355 10583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:29.725172 10583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab
--principal=oryx/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:29.726454 10583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:29.727753 10583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:29.740078 10589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:29.740151 10588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:29.740115 10591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:29.741308 10583 server_base.cc:1034] running on GCE node
I20260928 10:58:29.741952 10583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:29.742892 10583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:29.744078 10583 hybrid_clock.cc:648] HybridClock initialized: now 1790593109744038 us; error 33 us; skew 500 ppm
I20260928 10:58:29.744388 10583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:29 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593109, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:29.747354 10583 init.cc:377] Logged in from keytab as oryx/127.3.252.67@KRBTEST.COM (short username oryx)
I20260928 10:58:29.748699 10583 webserver.cc:460] Webserver started at http://127.3.252.67:40215/ using document root <none> and password file <none>
I20260928 10:58:29.749360 10583 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:29.749469 10583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:29.749747 10583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:29.752908 10583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "1cc9822af1d7416dac64358e3b58a664"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "0b2e62b7409e3fe57de8e2ec704df332"
server_key_iv: "543920d2c03a84ce06da08c98083ca61"
server_key_version: "encryptionkey@0"
I20260928 10:58:29.753620 10583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "1cc9822af1d7416dac64358e3b58a664"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "0b2e62b7409e3fe57de8e2ec704df332"
server_key_iv: "543920d2c03a84ce06da08c98083ca61"
server_key_version: "encryptionkey@0"
I20260928 10:58:29.758845 10583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:58:29.762174 10598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:29.763650 10583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 10:58:29.763820 10583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "1cc9822af1d7416dac64358e3b58a664"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "0b2e62b7409e3fe57de8e2ec704df332"
server_key_iv: "543920d2c03a84ce06da08c98083ca61"
server_key_version: "encryptionkey@0"
I20260928 10:58:29.764016 10583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:29.799274 10583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:29.803344 10583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:29.803622 10583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:29.804530 10583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:29.805864 10583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:29.805940 10583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:29.806061 10583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:29.806121 10583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:29.837625 10583 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:34109
I20260928 10:58:29.837673 10710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:34109 every 8 connection(s)
I20260928 10:58:29.838881 10583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:29.845955 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 10583
I20260928 10:58:29.846127 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:29.846546 4081 external_mini_cluster.cc:1428] Setting key 2104489d6ab415cf57c2c8c65a67d918
Sep 28 10:58:29 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593109, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:29.857097 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:29.841249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:47215 (local address 127.3.252.126:33045)
0928 10:58:29.841446 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:29.841450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:29.842488 (+ 1038us) server_negotiation.cc:407] Connection header received
0928 10:58:29.844260 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:29.844266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:29.844342 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:29.844430 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:29.845902 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.846723 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.848444 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.848872 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.848983 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:29.852212 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:29.852219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:29.852223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:29.852272 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:29.854689 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:29.855300 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:29.855303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:29.855306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:29.855416 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:29.855750 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:29.855753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:29.855755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:29.855993 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:29.856092 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:29.856708 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 10:58:29.856861 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:58:29.858196 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:29.841669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.67:47215)
0928 10:58:29.842332 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:29.842376 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:29.843994 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:29.844669 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:29.844677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:29.845158 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:29.845721 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:29.845739 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.846871 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.846875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:29.848320 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:29.848333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:29.849096 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:29.849104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:29.849317 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:29.849962 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:29.849989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:29.852018 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:29.854810 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:29.854816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:29.854829 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:29.855215 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:29.855496 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:29.855499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:29.855501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:29.855675 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:29.856078 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:29.856085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:29.856391 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 10:58:29.856643 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 10:58:29.857021 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":274,"threads_started":1}
I20260928 10:58:29.859562 10711 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33045
I20260928 10:58:29.859861 10711 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:29.860639 10711 heartbeater.cc:511] Master 127.3.252.126:33045 requested a full tablet report, sending...
I20260928 10:58:29.861982 10255 ts_manager.cc:194] Registered new tserver with Master: 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109)
I20260928 10:58:29.862754 10255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:47215
I20260928 10:58:29.862994 10444 heartbeater.cc:503] Master 127.3.252.126:33045 was elected leader, sending a full tablet report...
I20260928 10:58:29.873605 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.252.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.125.keytab.
Entry for principal oryx/127.3.252.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.125.keytab.
I20260928 10:58:29.916638 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:43305
--webserver_interface=127.3.252.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.125.keytab
--principal=oryx/127.3.252.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:33045,127.3.252.125:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:30.225162 10716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:30.225458 10716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:30.225596 10716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:30.232679 10716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:30.232800 10716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:30.232874 10716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:30.232944 10716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:30.233002 10716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:30.243152 10716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:33045,127.3.252.125:43305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.125.keytab
--principal=oryx/127.3.252.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.252.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
W20260928 10:58:30.243553 10716 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.252.126:33045,127.3.252.125:43305'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 10:58:30.244598 10716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:30.245929 10716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:30.257668 10721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:30.257644 10722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:30.258185 10724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:30.258443 10716 server_base.cc:1034] running on GCE node
I20260928 10:58:30.259191 10716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:30.260546 10716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:30.261776 10716 hybrid_clock.cc:648] HybridClock initialized: now 1790593110261711 us; error 45 us; skew 500 ppm
I20260928 10:58:30.262105 10716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:30.269341 10716 init.cc:377] Logged in from keytab as oryx/127.3.252.125@KRBTEST.COM (short username oryx)
I20260928 10:58:30.270784 10716 webserver.cc:460] Webserver started at http://127.3.252.125:39445/ using document root <none> and password file <none>
I20260928 10:58:30.271463 10716 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:30.271551 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:30.271855 10716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:30.274731 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/instance:
uuid: "f91d1398d5554f60a9b994ea9638d2fe"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "2398004193623344ec5c78c2058154f2"
server_key_iv: "48ed6078948dc027109441f95eedeb4f"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.275452 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal/instance:
uuid: "f91d1398d5554f60a9b994ea9638d2fe"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "2398004193623344ec5c78c2058154f2"
server_key_iv: "48ed6078948dc027109441f95eedeb4f"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.280383 10716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:30.283756 10731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:30.285053 10716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:30.285233 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
uuid: "f91d1398d5554f60a9b994ea9638d2fe"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "2398004193623344ec5c78c2058154f2"
server_key_iv: "48ed6078948dc027109441f95eedeb4f"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.285411 10716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:30.310949 10716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:30.315807 10716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:30.316200 10716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:30.335896 10716 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.125:43305
I20260928 10:58:30.335903 10782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.125:43305 every 8 connection(s)
I20260928 10:58:30.337280 10716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
I20260928 10:58:30.341184 10783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:30.344962 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 10716
I20260928 10:58:30.348277 10783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.125@KRBTEST.COM
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.125@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:30.367013 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.352351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.3.252.126:33045)
0928 10:58:30.352546 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.352553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.353228 (+ 675us) server_negotiation.cc:407] Connection header received
0928 10:58:30.353407 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.353411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.353472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.353549 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.354897 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.355586 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.357029 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.357452 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.357569 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.361326 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:30.361331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:30.361334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:30.361371 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:30.364118 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.364910 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.364916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.364918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.365045 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.365426 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.365429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.365431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.365755 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:30.365906 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:30.366457 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 10:58:30.366699 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:58:30.368067 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.352311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:55708)
0928 10:58:30.353061 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:30.353119 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:30.353293 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:30.353635 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:30.353642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:30.354466 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:30.354760 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.354768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.355726 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.355730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.356891 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.356903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.357676 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.357683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.357858 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.358420 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:30.358443 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:30.361023 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:30.364293 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.364307 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.364320 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.364783 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.365158 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.365163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.365167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.365321 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.365894 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:30.365903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:30.366105 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 10:58:30.366364 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 10:58:30.366902 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":5,"thread_start_us":348,"threads_started":1}
I20260928 10:58:30.368783 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.347188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.3.252.125:43305)
0928 10:58:30.347987 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.347999 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.348040 (+ 41us) server_negotiation.cc:407] Connection header received
0928 10:58:30.349730 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.349756 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.350253 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.350639 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.352154 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.354458 (+ 2304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.355780 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.356527 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.356726 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.360941 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:30.360951 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:30.360965 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:30.360999 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:30.363955 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.364751 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.364758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.364761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.364914 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.365347 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.365352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.365356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.365963 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:30.366246 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:30.366570 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 10:58:30.368516 (+ 1946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1749,"server-negotiator.queue_time_us":606,"thread_start_us":408,"threads_started":1}
I20260928 10:58:30.369994 10783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:58:30.370596 10578 heartbeater.cc:503] Master 127.3.252.126:33045 was elected leader, sending a full tablet report...
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.125@KRBTEST.COM for oryx/127.3.252.125@KRBTEST.COM
I20260928 10:58:30.382447 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.370954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.3.252.125:43305)
0928 10:58:30.371089 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.371094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.371111 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:58:30.371157 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.371161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.371242 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.371653 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.372112 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.372833 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.374048 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.374461 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.374557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.377453 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:30.377458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:30.377461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:30.377490 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:30.379877 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.380531 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.380536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.380538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.380692 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.381112 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.381116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.381119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.381420 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:30.381533 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:30.381944 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 10:58:30.382202 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":48}
I20260928 10:58:30.382495 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.370749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.0.0.1:50384)
0928 10:58:30.370939 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:30.370958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:30.371054 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:30.371377 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:30.371382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:30.371818 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:30.371976 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.371989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.372988 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.372992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.373916 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.373928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.374598 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.374602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.374751 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.375253 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:30.375270 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:30.377275 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:30.380033 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.380037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.380040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.380407 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.380820 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.380824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.380826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.380964 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.381543 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:30.381546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:30.381671 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 10:58:30.381784 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 10:58:30.382077 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 10:58:30.387272 10783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Bootstrap starting.
I20260928 10:58:30.390414 10783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:30.391419 10783 log.cc:824] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:30.394018 10783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: No bootstrap required, opened a new log
I20260928 10:58:30.401422 10783 raft_consensus.cc:348] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } }
I20260928 10:58:30.401767 10783 raft_consensus.cc:374] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:30.401866 10783 raft_consensus.cc:729] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91d1398d5554f60a9b994ea9638d2fe, State: Initialized, Role: FOLLOWER
I20260928 10:58:30.402450 10783 consensus_queue.cc:260] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } }
I20260928 10:58:30.403246 10789 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } } }
I20260928 10:58:30.403556 10789 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:58:30.404206 10783 sys_catalog.cc:564] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: configured and running, proceeding with master startup.
W20260928 10:58:30.412895 10800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:58:30.413051 10800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:58:30.415385 10716 master_runner.cc:428] Detected that this master f91d1398d5554f60a9b994ea9638d2fe is joining an existing cluster
I20260928 10:58:30.415490 10716 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.252.125:43305
I20260928 10:58:30.435316 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.419637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:55716)
0928 10:58:30.420314 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:30.420338 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:30.420566 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:30.420979 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:30.420983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:30.421719 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:30.422409 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.422423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.423929 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.423934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.425641 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.425655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.426845 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.426851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.427018 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.427681 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:30.427744 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:30.428950 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:30.432640 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.432646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.432654 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.433099 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.433462 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.433467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.433471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.433665 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.434312 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:30.434326 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:30.434538 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 10:58:30.434729 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 10:58:30.434997 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"spinlock_wait_cycles":3840,"thread_start_us":352,"threads_started":1}
I20260928 10:58:30.435478 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.419637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.3.252.126:33045)
0928 10:58:30.419838 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.419845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.420446 (+ 601us) server_negotiation.cc:407] Connection header received
0928 10:58:30.420747 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.420753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.420844 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.420958 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.422599 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.423747 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.425811 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.426461 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.426563 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.429194 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:30.429201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:30.429206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:30.429249 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:30.432471 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.433214 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.433220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.433224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.433356 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.433795 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.433799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.433802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.434128 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:30.434225 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:30.434903 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 10:58:30.435139 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:58:30.437126 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.420469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.3.252.125:43305)
0928 10:58:30.420613 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.420618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.421693 (+ 1075us) server_negotiation.cc:407] Connection header received
0928 10:58:30.421800 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.421805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.421893 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.421977 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.423227 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.424301 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.426172 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.426799 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.426880 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.429435 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:30.429441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:30.429445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:30.429474 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:30.432840 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.433663 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.433668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.433672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.433820 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.434259 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.434264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.434267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.434650 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:30.434777 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:30.436412 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0928 10:58:30.436779 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":45}
I20260928 10:58:30.437099 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.420551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.0.0.1:50398)
0928 10:58:30.421579 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:30.421594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:30.421702 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:30.422149 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:30.422153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:30.422756 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:30.423092 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.423104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.424536 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.424541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.426022 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.426035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.427138 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.427143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.427603 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.428230 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:30.428249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:30.429285 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:30.432978 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.432984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.432988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.433532 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.433972 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.433976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.433979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.434136 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.435984 (+ 1848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:30.435989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:30.436155 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 10:58:30.436299 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 10:58:30.436599 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"spinlock_wait_cycles":35456,"thread_start_us":764,"threads_started":1}
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593108, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.126@KRBTEST.COM for oryx/127.3.252.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.252.124@KRBTEST.COM; defaulting to no policy
I20260928 10:58:30.455560 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.441600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.3.252.125:43305)
0928 10:58:30.441764 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.441771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.442221 (+ 450us) server_negotiation.cc:407] Connection header received
0928 10:58:30.442470 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.442475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.442553 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.442639 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.443674 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.444333 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.445618 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.446155 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.446275 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.449294 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:30.449299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:30.449302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:30.449333 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:30.452330 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.453073 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.453079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.453083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.453221 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.454041 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.454047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.454050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.454360 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:30.454493 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:30.455034 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 10:58:30.455322 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:58:30.455750 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.441510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.0.0.1:50408)
0928 10:58:30.442097 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:30.442139 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:30.442300 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:30.442707 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:30.442714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:30.443302 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:30.443527 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.443537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.444459 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.444465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.445489 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.445501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.446307 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.446313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.446492 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.447098 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:30.447115 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:30.449111 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:30.452500 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.452515 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.452522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.452958 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.453325 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.453328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.453331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.453485 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.454539 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:30.454548 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:30.454748 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 10:58:30.454968 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 10:58:30.455317 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":256,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.124@KRBTEST.COM" created.
I20260928 10:58:30.458001 10255 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.252.125:43305: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe"
member_type: NON_VOTER
last_known_addr {
host: "127.3.252.125"
port: 43305
}
attrs {
promote: true
}
}
dest_uuid: "7f1cb973abd94f638b2275484463e071"
cas_config_opid_index: -1
I20260928 10:58:30.459110 10255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } }
I20260928 10:58:30.468045 10811 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f91d1398d5554f60a9b994ea9638d2fe (127.3.252.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } }
I20260928 10:58:30.469498 10758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:30.472586 10811 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.252.125:43305
I20260928 10:58:30.473063 10758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Refusing update from remote peer 7f1cb973abd94f638b2275484463e071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 10:58:30.473317 10716 master.cc:537] Master@127.3.252.125:43305 shutting down...
I20260928 10:58:30.474416 10811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:30.474740 10812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } }
I20260928 10:58:30.475064 10812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:30.489861 10716 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:30.490326 10716 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:30.490531 10716 tablet_replica.cc:330] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: stopping tablet replica
W20260928 10:58:30.490573 10239 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 10:58:30.494004 10239 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 -> Peer f91d1398d5554f60a9b994ea9638d2fe (127.3.252.125:43305): Couldn't send request to peer f91d1398d5554f60a9b994ea9638d2fe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.124.keytab.
Entry for principal oryx/127.3.252.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.124.keytab.
I20260928 10:58:30.501940 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:44755
--webserver_interface=127.3.252.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.124.keytab
--principal=oryx/127.3.252.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:58:30.507181 10716 master.cc:552] Master@127.3.252.125:43305 shutdown complete.
I20260928 10:58:30.507266 10716 master_runner.cc:305] Clearing existing system tablet
I20260928 10:58:30.507639 10716 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:30.507726 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:30.510282 10716 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 10:58:30.511829 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:30.512166 10716 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:30.512400 10716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 10:58:30.512470 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
uuid: "f91d1398d5554f60a9b994ea9638d2fe"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "2398004193623344ec5c78c2058154f2"
server_key_iv: "48ed6078948dc027109441f95eedeb4f"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.516979 10716 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:30.523748 10716 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 10:58:30.523947 10716 log.cc:1196] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 10:58:30.524403 10716 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Deleting consensus metadata
I20260928 10:58:30.524720 10716 master_runner.cc:315] Copying system tablet from 127.3.252.126:33045
I20260928 10:58:30.529707 10716 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: tablet copy: Beginning tablet copy session from remote peer at address 127.3.252.126:33045
I20260928 10:58:30.547909 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.530561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.3.252.126:33045)
0928 10:58:30.530741 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.530747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.534422 (+ 3675us) server_negotiation.cc:407] Connection header received
0928 10:58:30.534684 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.534691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.534787 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.534901 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.535938 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.537042 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.538858 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.539477 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.539619 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.541787 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:30.541794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:30.541799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:30.541874 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:30.545102 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.545883 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.545888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.545892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.546040 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:30.546427 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:30.546432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:30.546435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:30.546788 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:30.546941 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:30.547346 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 10:58:30.547575 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:58:30.547969 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.530500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:55732)
0928 10:58:30.534265 (+ 3765us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:30.534293 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:30.534477 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:30.534941 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:30.534946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:30.535450 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:30.535763 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.535776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.537246 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.537252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.538700 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.538715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.539656 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.539662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.539822 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.540527 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:30.540557 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:30.541562 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:30.545250 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.545256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.545260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.545747 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.546151 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:30.546155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:30.546158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:30.546327 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:30.546907 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:30.546912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:30.547088 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 10:58:30.547241 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 10:58:30.547557 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3622,"thread_start_us":3487,"threads_started":1}
I20260928 10:58:30.549073 10275 tablet_copy_service.cc:140] P 7f1cb973abd94f638b2275484463e071: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f91d1398d5554f60a9b994ea9638d2fe ({username='oryx', principal='oryx/127.3.252.125@KRBTEST.COM'} at 127.0.0.1:55732)
I20260928 10:58:30.549353 10275 tablet_copy_service.cc:161] P 7f1cb973abd94f638b2275484463e071: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f91d1398d5554f60a9b994ea9638d2fe at {username='oryx', principal='oryx/127.3.252.125@KRBTEST.COM'} at 127.0.0.1:55732: session id = f91d1398d5554f60a9b994ea9638d2fe-00000000000000000000000000000000
I20260928 10:58:30.553785 10275 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 10:58:30.555989 10716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 10:58:30.563136 10716 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: tablet copy: Starting download of 0 data blocks...
I20260928 10:58:30.563555 10716 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: tablet copy: Starting download of 1 WAL segments...
I20260928 10:58:30.566498 10716 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 10:58:30.570892 10275 tablet_copy_service.cc:342] P 7f1cb973abd94f638b2275484463e071: Request end of tablet copy session f91d1398d5554f60a9b994ea9638d2fe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.252.125@KRBTEST.COM'} at 127.0.0.1:55732
I20260928 10:58:30.571108 10275 tablet_copy_service.cc:433] P 7f1cb973abd94f638b2275484463e071: ending tablet copy session f91d1398d5554f60a9b994ea9638d2fe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f91d1398d5554f60a9b994ea9638d2fe
I20260928 10:58:30.582710 10716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:30.585399 10826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:30.590669 10827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:30.598065 10829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:30.603343 10716 server_base.cc:1034] running on GCE node
I20260928 10:58:30.603735 10716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:30.604377 10716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:30.605592 10716 hybrid_clock.cc:648] HybridClock initialized: now 1790593110605520 us; error 41 us; skew 500 ppm
I20260928 10:58:30.605818 10716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:30.607909 10716 init.cc:377] Logged in from keytab as oryx/127.3.252.125@KRBTEST.COM (short username oryx)
I20260928 10:58:30.609148 10716 webserver.cc:460] Webserver started at http://127.3.252.125:40279/ using document root <none> and password file <none>
I20260928 10:58:30.609486 10716 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:30.609570 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:30.612540 10716 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 10:58:30.614398 10836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:30.614840 10716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 10:58:30.614976 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
uuid: "f91d1398d5554f60a9b994ea9638d2fe"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "2398004193623344ec5c78c2058154f2"
server_key_iv: "48ed6078948dc027109441f95eedeb4f"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.615152 10716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:30.638121 10716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:30.641938 10716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:30.668613 10716 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.125:43305
I20260928 10:58:30.668646 10887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.125:43305 every 8 connection(s)
I20260928 10:58:30.669637 10716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
I20260928 10:58:30.674817 10888 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:58:30.678529 10888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Bootstrap starting.
I20260928 10:58:30.695454 10888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:58:30.696056 10888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Bootstrap complete.
I20260928 10:58:30.697242 10888 raft_consensus.cc:348] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } }
I20260928 10:58:30.697552 10888 raft_consensus.cc:729] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f91d1398d5554f60a9b994ea9638d2fe, State: Initialized, Role: LEARNER
I20260928 10:58:30.697923 10888 consensus_queue.cc:260] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } }
I20260928 10:58:30.698837 10891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } }
I20260928 10:58:30.699178 10891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: LEARNER
I20260928 10:58:30.699301 10888 sys_catalog.cc:564] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:30.713600 10902 catalog_manager.cc:1259] Loaded cluster ID: b7a8d97379564ff68befb672b741c40a
I20260928 10:58:30.713718 10902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: loading cluster ID for follower catalog manager: success
I20260928 10:58:30.717147 10902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: acquiring CA information for follower catalog manager: success
I20260928 10:58:30.719549 10902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:58:30.865290 10711 heartbeater.cc:503] Master 127.3.252.126:33045 was elected leader, sending a full tablet report...
W20260928 10:58:30.874163 10817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:30.874455 10817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:30.874589 10817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:30.881680 10817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:30.881791 10817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:30.881863 10817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:30.881942 10817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:30.881989 10817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:30.892434 10817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.124.keytab
--principal=oryx/127.3.252.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:44755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.252.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:30.893913 10817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:30.895278 10817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:30.906792 10907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:30.906890 10910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:30.907173 10908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:30.907655 10817 server_base.cc:1034] running on GCE node
I20260928 10:58:30.908386 10817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:30.909742 10817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:30.910928 10817 hybrid_clock.cc:648] HybridClock initialized: now 1790593110910886 us; error 45 us; skew 500 ppm
I20260928 10:58:30.911242 10817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:30.914127 10817 init.cc:377] Logged in from keytab as oryx/127.3.252.124@KRBTEST.COM (short username oryx)
I20260928 10:58:30.915433 10817 webserver.cc:460] Webserver started at http://127.3.252.124:46221/ using document root <none> and password file <none>
I20260928 10:58:30.916103 10817 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:30.916206 10817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:30.916471 10817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:30.919323 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/instance:
uuid: "203df8740c5643aa8a941b846bb92d01"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "d17f3338066d7c0331ec4a9a7656e5ec"
server_key_iv: "b44325b54c5ddb7824e0a9ff3fb9be18"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.920045 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal/instance:
uuid: "203df8740c5643aa8a941b846bb92d01"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "d17f3338066d7c0331ec4a9a7656e5ec"
server_key_iv: "b44325b54c5ddb7824e0a9ff3fb9be18"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.925292 10817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 10:58:30.928514 10917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:30.929744 10817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:30.929914 10817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
uuid: "203df8740c5643aa8a941b846bb92d01"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "d17f3338066d7c0331ec4a9a7656e5ec"
server_key_iv: "b44325b54c5ddb7824e0a9ff3fb9be18"
server_key_version: "encryptionkey@0"
I20260928 10:58:30.930080 10817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:30.953433 10817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:30.957413 10817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:30.957662 10817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:30.975500 10817 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.124:44755
I20260928 10:58:30.975533 10968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.124:44755 every 8 connection(s)
I20260928 10:58:30.976769 10817 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
I20260928 10:58:30.980238 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:30.986326 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 10817
I20260928 10:58:30.987280 10969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.124@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
Sep 28 10:58:30 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.124@KRBTEST.COM
I20260928 10:58:31.006608 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.990127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.3.252.126:33045)
0928 10:58:30.990283 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.990288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.990961 (+ 673us) server_negotiation.cc:407] Connection header received
0928 10:58:30.991815 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.991821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.991895 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.991975 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.993176 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.994310 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.996290 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.996672 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.996793 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.000747 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.000752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.000755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.000788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.003644 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.004399 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.004404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.004407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.004516 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.004918 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.004924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.004927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.005247 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.005369 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.006081 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.006302 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:58:31.007668 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.990140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:55742)
0928 10:58:30.990831 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:30.990863 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:30.991590 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:30.992056 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:30.992063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:30.992684 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:30.993024 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.993040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.994476 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.994480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.996134 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:30.996148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.996876 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.996881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:30.997138 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:30.997812 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:30.997872 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.000574 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.003760 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.003771 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.003787 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.004276 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.004619 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.004624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.004627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.004818 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.005356 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.005363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.005697 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.005988 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.006464 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":46,"thread_start_us":354,"threads_started":1}
I20260928 10:58:31.008261 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:30.988572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.3.252.124:44755)
0928 10:58:30.989350 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:30.989360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:30.989398 (+ 38us) server_negotiation.cc:407] Connection header received
0928 10:58:30.991586 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:30.991606 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:30.991861 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:30.992034 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:30.993490 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.995047 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.996707 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:30.997350 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:30.997446 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.002677 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.002685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.002699 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.002734 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.005376 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.006062 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.006069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.006071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.006190 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.006554 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.006558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.006560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.006981 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.007276 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.007593 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.007816 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":575,"thread_start_us":356,"threads_started":1}
I20260928 10:58:31.009274 10969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.124@KRBTEST.COM for oryx/127.3.252.125@KRBTEST.COM
I20260928 10:58:31.021327 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.009869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.0.0.1:50424)
0928 10:58:31.010065 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.010078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.010150 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.010802 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.010807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.011133 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.011258 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.011265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.012469 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.012474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.013380 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.013390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.014085 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.014090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.014295 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.014832 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.014851 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.016719 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.019433 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.019436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.019439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.019764 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.020070 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.020073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.020075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.020184 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.020633 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.020636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.020762 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.020860 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.021057 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 10:58:31.021351 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.010065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.3.252.125:43305)
0928 10:58:31.010543 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.010547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.010562 (+ 15us) server_negotiation.cc:407] Connection header received
0928 10:58:31.010606 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.010610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.010667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.010754 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.011421 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.012353 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.013502 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.013960 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.014066 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.016885 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.016891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.016894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.016926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.019333 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.019846 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.019849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.019852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.019970 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.020259 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.020262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.020264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.020518 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.020672 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.020930 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.021085 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
I20260928 10:58:31.022390 10969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593110, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.124@KRBTEST.COM for oryx/127.3.252.124@KRBTEST.COM
I20260928 10:58:31.033898 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.023202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.0.0.1:56348)
0928 10:58:31.023369 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.023387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.023495 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.023749 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.023753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.024087 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.024217 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.024225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.025117 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.025121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.025971 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.025981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.026624 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.026630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.026827 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.027330 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.027347 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.029193 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.031721 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.031727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.031731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.032056 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.032357 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.032361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.032365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.032498 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.032936 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.032940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.033077 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.033213 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.033462 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 10:58:31.033905 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.023277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.3.252.124:44755)
0928 10:58:31.023407 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.023412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.023429 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:58:31.023595 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.023600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.023655 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.023724 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.024332 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.025010 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.026097 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.026502 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.026605 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.029350 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.029355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.029358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.029383 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.031614 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.032150 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.032156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.032159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.032264 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.032588 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.032593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.032596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.032838 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.032953 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.033305 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.033590 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":55}
I20260928 10:58:31.039230 10969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Bootstrap starting.
I20260928 10:58:31.042385 10969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:31.043367 10969 log.cc:824] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:31.046154 10969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: No bootstrap required, opened a new log
I20260928 10:58:31.053416 10969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } }
I20260928 10:58:31.053798 10969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:31.053933 10969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203df8740c5643aa8a941b846bb92d01, State: Initialized, Role: FOLLOWER
I20260928 10:58:31.054488 10969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } }
I20260928 10:58:31.055298 10976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } } }
I20260928 10:58:31.055624 10976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:58:31.056331 10969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:31.063007 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 10316
W20260928 10:58:31.066268 10987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:58:31.066427 10987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:58:31.069327 10817 master_runner.cc:428] Detected that this master 203df8740c5643aa8a941b846bb92d01 is joining an existing cluster
I20260928 10:58:31.069429 10817 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.252.124:44755
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.65@KRBTEST.COM
I20260928 10:58:31.089219 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.064666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.3.252.65:42053)
0928 10:58:31.066966 (+ 2300us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.066974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.067029 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:58:31.067096 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.067104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.067343 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.067503 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.069179 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.070802 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.073210 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.074549 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.080260 (+ 5711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.080565 (+ 305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.080579 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.080597 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.080632 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.084899 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.085766 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.085775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.085779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.085950 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.086483 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.086488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.086491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.086933 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.088350 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.088523 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.088795 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2139,"thread_start_us":1986,"threads_started":1}
I20260928 10:58:31.090724 10439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:31.106137 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.073736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.3.252.126:33045)
0928 10:58:31.073958 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.073963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.086761 (+ 12798us) server_negotiation.cc:407] Connection header received
0928 10:58:31.088388 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.088395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.088487 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.088556 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.091764 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.092988 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.095547 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.096176 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.096278 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.098994 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.099001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.099006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.099049 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.102368 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.103244 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.103250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.103254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.103410 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.103944 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.103953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.103956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.104304 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.104432 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.105240 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.105725 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 10:58:31.107106 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.073788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:55752)
0928 10:58:31.086628 (+ 12840us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.086652 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.088198 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.088780 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.088785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.090397 (+ 1612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.090793 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.090807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.093134 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.093139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.095366 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.095381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.096586 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.096592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.096805 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.097505 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.097538 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.098734 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.102549 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.102553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.102561 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.103108 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.103585 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.103590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.103593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.103783 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.104509 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.104513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.104926 (+ 413us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.106330 (+ 1404us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.106666 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12660,"thread_start_us":12973,"threads_started":1}
I20260928 10:58:31.117286 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.087586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.3.252.125:43305)
0928 10:58:31.087776 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.087782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.093261 (+ 5479us) server_negotiation.cc:407] Connection header received
0928 10:58:31.093382 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.093388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.093503 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.093597 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.099178 (+ 5581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.100360 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.105308 (+ 4948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.106093 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.106190 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.109644 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.109652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.109657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.109699 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.112868 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.114976 (+ 2108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.114982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.114986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.115150 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.115616 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.115621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.115625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.115984 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.116114 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.116719 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.116945 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:58:31.118748 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.087930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.0.0.1:50434)
0928 10:58:31.093134 (+ 5204us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.093150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.093270 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.093817 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.093823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.095545 (+ 1722us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.095876 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.095888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.100631 (+ 4743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.100637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.103489 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.103503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.107547 (+ 4044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.107554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.107730 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.108290 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.108312 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.109405 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.113028 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.113033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.113037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.114801 (+ 1764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.115296 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.115300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.115304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.115477 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.116217 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.116222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.116425 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.117511 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.118298 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5102,"thread_start_us":4567,"threads_started":1}
I20260928 10:58:31.129248 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.074177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.3.252.124:44755)
0928 10:58:31.074345 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.074351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.092493 (+ 18142us) server_negotiation.cc:407] Connection header received
0928 10:58:31.093428 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.093433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.093513 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.093572 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.096854 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.098057 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.103650 (+ 5593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.104413 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.104507 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.109273 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.109280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.109284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.109326 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.114479 (+ 5153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.127062 (+ 12583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.127068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.127072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.127212 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.127608 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.127613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.127616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.128025 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.128160 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.128640 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.128881 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":57}
I20260928 10:58:31.130287 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.074072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.0.0.1:56354)
0928 10:58:31.092404 (+ 18332us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.092426 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.092715 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.093787 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.093792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.096335 (+ 2543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.096702 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.096714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.098287 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.098296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.102700 (+ 4404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.102716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.104669 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.104675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.104816 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.107088 (+ 2272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.107115 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.109077 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.126329 (+ 17252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.126334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.126339 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.126918 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.127337 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.127342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.127345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.127474 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.128222 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.128226 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.128385 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.129522 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.129804 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18179,"thread_start_us":18823,"threads_started":1}
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593108, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.126@KRBTEST.COM for oryx/127.3.252.124@KRBTEST.COM
I20260928 10:58:31.143728 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.135373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.3.252.125:43305)
0928 10:58:31.135547 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.135552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.135819 (+ 267us) server_negotiation.cc:407] Connection header received
0928 10:58:31.136067 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.136073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.136159 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.136233 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:31.136927 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.138002 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.140544 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.141873 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.141981 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.142217 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.143288 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.143436 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:58:31.144826 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.135191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.0.0.1:50438)
0928 10:58:31.135715 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.135737 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.135915 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.136417 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.136421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.136446 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:31.136781 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.136791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.138163 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.138168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.140358 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.140374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.142491 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.142497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.143071 (+ 574us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.143141 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.144082 (+ 941us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.144237 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":256,"threads_started":1}
I20260928 10:58:31.146090 10863 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 10:58:31.146836 10891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1cb973abd94f638b2275484463e071. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } }
I20260928 10:58:31.147080 10891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: LEARNER
I20260928 10:58:31.149917 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.114649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.0.0.1:56358)
0928 10:58:31.131688 (+ 17039us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.131710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.131879 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.132447 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.132452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.132965 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.133261 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.133275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.135637 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.135642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.137064 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.137079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.138332 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.138339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.138534 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.139240 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.139265 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.142309 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.146947 (+ 4638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.146953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.146957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.147510 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.148002 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.148007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.148011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.148177 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.148856 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.148861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.149050 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.149193 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.149478 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16915,"thread_start_us":16803,"threads_started":1}
I20260928 10:58:31.150207 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.114943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.3.252.124:44755)
0928 10:58:31.129488 (+ 14545us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.129494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.131769 (+ 2275us) server_negotiation.cc:407] Connection header received
0928 10:58:31.132042 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.132047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.132146 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.132212 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.133956 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.135064 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.137223 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.137861 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.137955 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.142571 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.142579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.142583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.142634 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.146769 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.147653 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.147658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.147662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.147829 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.148323 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.148329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.148333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.148692 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.148834 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.149321 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.149589 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14448,"thread_start_us":16242,"threads_started":1}
I20260928 10:58:31.151847 10254 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.252.124:44755: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "203df8740c5643aa8a941b846bb92d01"
member_type: NON_VOTER
last_known_addr {
host: "127.3.252.124"
port: 44755
}
attrs {
promote: true
}
}
dest_uuid: "7f1cb973abd94f638b2275484463e071"
cas_config_opid_index: 5
I20260928 10:58:31.152858 10254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } }
I20260928 10:58:31.154635 10998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 203df8740c5643aa8a941b846bb92d01 (127.3.252.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.155946 10944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:31.158196 10998 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.252.124:44755
I20260928 10:58:31.159089 10863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 LEARNER]: Refusing update from remote peer 7f1cb973abd94f638b2275484463e071: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 10:58:31.160162 10817 master.cc:537] Master@127.3.252.124:44755 shutting down...
I20260928 10:58:31.162892 10944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Refusing update from remote peer 7f1cb973abd94f638b2275484463e071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 10:58:31.163650 11002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 10:58:31.164330 11002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:31.172986 11003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.173424 11003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:31.181427 10863 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 203df8740c5643aa8a941b846bb92d01 (127.3.252.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
W20260928 10:58:31.182610 10242 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 -> Peer 203df8740c5643aa8a941b846bb92d01 (127.3.252.124:44755): Couldn't send request to peer 203df8740c5643aa8a941b846bb92d01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 10:58:31.186005 10891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: NON_VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: true } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.186282 10891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: LEARNER
I20260928 10:58:31.187166 11003 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: attempting to promote NON_VOTER f91d1398d5554f60a9b994ea9638d2fe to VOTER
I20260928 10:58:31.188086 11003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } }
I20260928 10:58:31.190614 10863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 LEARNER]: Refusing update from remote peer 7f1cb973abd94f638b2275484463e071: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260928 10:58:31.190896 10242 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 -> Peer 203df8740c5643aa8a941b846bb92d01 (127.3.252.124:44755): Couldn't send request to peer 203df8740c5643aa8a941b846bb92d01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 10:58:31.191183 11003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 10:58:31.193404 11002 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f91d1398d5554f60a9b994ea9638d2fe (127.3.252.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.194216 10863 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f91d1398d5554f60a9b994ea9638d2fe (127.3.252.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.197506 11003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.197573 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.197791 11003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:31.198678 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:31.199301 10891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.199537 10891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:58:31.200380 10817 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:31.200798 10817 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:31.201009 10817 tablet_replica.cc:330] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: stopping tablet replica
I20260928 10:58:31.217751 10817 master.cc:552] Master@127.3.252.124:44755 shutdown complete.
I20260928 10:58:31.217872 10817 master_runner.cc:305] Clearing existing system tablet
I20260928 10:58:31.218262 10817 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:31.218377 10817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:31.221412 10817 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 10:58:31.223070 11008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:31.223368 10817 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:31.223640 10817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 10:58:31.223742 10817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
uuid: "203df8740c5643aa8a941b846bb92d01"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "d17f3338066d7c0331ec4a9a7656e5ec"
server_key_iv: "b44325b54c5ddb7824e0a9ff3fb9be18"
server_key_version: "encryptionkey@0"
I20260928 10:58:31.229040 10817 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:31.231194 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 10316
I20260928 10:58:31.236131 10817 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 10:58:31.236311 10817 log.cc:1196] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 10:58:31.236776 10817 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Deleting consensus metadata
I20260928 10:58:31.237044 10817 master_runner.cc:315] Copying system tablet from 127.3.252.126:33045
I20260928 10:58:31.241626 10817 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: tablet copy: Beginning tablet copy session from remote peer at address 127.3.252.126:33045
I20260928 10:58:31.248898 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:42053
--local_ip_for_outbound_sockets=127.3.252.65
--tserver_master_addrs=127.3.252.126:33045
--webserver_port=45997
--webserver_interface=127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab
--principal=oryx/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:58:31.259892 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.242429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:55768)
0928 10:58:31.246286 (+ 3857us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.246309 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.246482 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.246853 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.246858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.247420 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.247691 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.247704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.248840 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.248844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.250094 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.250110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.251306 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.251312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.251522 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.252183 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.252206 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.253348 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.257101 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.257107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.257111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.257600 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.258033 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.258041 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.258045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.258198 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.258854 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.258859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.259033 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.259167 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.259463 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3712,"thread_start_us":3576,"threads_started":1}
I20260928 10:58:31.259989 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.242495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.3.252.126:33045)
0928 10:58:31.242725 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.242732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.246428 (+ 3696us) server_negotiation.cc:407] Connection header received
0928 10:58:31.246610 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.246615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.246708 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.246808 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.247860 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.248651 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.250264 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.250929 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.251032 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.253581 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.253588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.253593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.253631 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.256948 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.257724 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.257730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.257733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.257903 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.258302 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.258307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.258310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.258656 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.258767 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.259340 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.259579 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 10:58:31.260471 10275 tablet_copy_service.cc:140] P 7f1cb973abd94f638b2275484463e071: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 203df8740c5643aa8a941b846bb92d01 ({username='oryx', principal='oryx/127.3.252.124@KRBTEST.COM'} at 127.0.0.1:55768)
I20260928 10:58:31.260691 10275 tablet_copy_service.cc:161] P 7f1cb973abd94f638b2275484463e071: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 203df8740c5643aa8a941b846bb92d01 at {username='oryx', principal='oryx/127.3.252.124@KRBTEST.COM'} at 127.0.0.1:55768: session id = 203df8740c5643aa8a941b846bb92d01-00000000000000000000000000000000
I20260928 10:58:31.263161 10275 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 10:58:31.265101 10817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 10:58:31.272464 10817 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: tablet copy: Starting download of 0 data blocks...
I20260928 10:58:31.272747 10817 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: tablet copy: Starting download of 1 WAL segments...
I20260928 10:58:31.275266 10817 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 10:58:31.279630 10275 tablet_copy_service.cc:342] P 7f1cb973abd94f638b2275484463e071: Request end of tablet copy session 203df8740c5643aa8a941b846bb92d01-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.252.124@KRBTEST.COM'} at 127.0.0.1:55768
I20260928 10:58:31.279776 10275 tablet_copy_service.cc:433] P 7f1cb973abd94f638b2275484463e071: ending tablet copy session 203df8740c5643aa8a941b846bb92d01-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 203df8740c5643aa8a941b846bb92d01
I20260928 10:58:31.292985 10817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:31.295761 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:31.299189 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:31.311996 11019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:31.312597 10817 server_base.cc:1034] running on GCE node
I20260928 10:58:31.313016 10817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:31.313915 10817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:31.315101 10817 hybrid_clock.cc:648] HybridClock initialized: now 1790593111315062 us; error 37 us; skew 500 ppm
I20260928 10:58:31.315380 10817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593111, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:31.317255 10817 init.cc:377] Logged in from keytab as oryx/127.3.252.124@KRBTEST.COM (short username oryx)
I20260928 10:58:31.318503 10817 webserver.cc:460] Webserver started at http://127.3.252.124:36005/ using document root <none> and password file <none>
I20260928 10:58:31.318763 10817 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:31.318846 10817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:31.321242 10817 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:31.322830 11026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:31.323221 10817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 10:58:31.323336 10817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
uuid: "203df8740c5643aa8a941b846bb92d01"
format_stamp: "Formatted at 2026-09-28 10:58:30 on dist-test-slave-djjw"
server_key: "d17f3338066d7c0331ec4a9a7656e5ec"
server_key_iv: "b44325b54c5ddb7824e0a9ff3fb9be18"
server_key_version: "encryptionkey@0"
I20260928 10:58:31.323523 10817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:31.345435 10817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:31.348516 10817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:31.366019 10817 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.124:44755
I20260928 10:58:31.366039 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.124:44755 every 8 connection(s)
I20260928 10:58:31.366983 10817 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
I20260928 10:58:31.372443 11078 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:58:31.376358 11078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Bootstrap starting.
I20260928 10:58:31.392246 11078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:58:31.392673 11078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: Bootstrap complete.
I20260928 10:58:31.393522 11078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } }
I20260928 10:58:31.393772 11078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 203df8740c5643aa8a941b846bb92d01, State: Initialized, Role: LEARNER
I20260928 10:58:31.394057 11078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } }
I20260928 10:58:31.395138 11081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.395435 11078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:31.395601 11081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:58:31.418550 11092 catalog_manager.cc:1259] Loaded cluster ID: b7a8d97379564ff68befb672b741c40a
I20260928 10:58:31.418660 11092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: loading cluster ID for follower catalog manager: success
I20260928 10:58:31.421319 11092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: acquiring CA information for follower catalog manager: success
I20260928 10:58:31.423164 11092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:58:31.592818 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.586195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.3.252.124:44755)
0928 10:58:31.586722 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.586727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.586743 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:58:31.586800 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.586803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.586868 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.586963 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:31.587421 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.588676 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.590924 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.591909 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.592029 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.592202 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.592446 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.592563 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":296,"threads_started":1}
I20260928 10:58:31.592885 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.586049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.0.0.1:56372)
0928 10:58:31.586288 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.586311 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.586452 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.587023 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.587028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.587057 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:31.587246 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.587258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.588836 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.588842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.590747 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.590765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.592087 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.592094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.592295 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.592331 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.592432 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.592578 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 10:58:31.594074 11053 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 10:58:31.594689 11081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1cb973abd94f638b2275484463e071. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:31.594964 11081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: This master's current role is: LEARNER
W20260928 10:58:31.639950 11015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:31.640244 11015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:31.640398 11015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:31.647456 11015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:31.647573 11015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:31.647732 11015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:31.657758 11015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab
--principal=oryx/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:42053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=45997
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:31.659487 11015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:31.660784 11015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:31.673661 11099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:31.673753 11098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:31.674391 11101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:31.675371 11015 server_base.cc:1034] running on GCE node
I20260928 10:58:31.676083 11015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:31.677201 11015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:31.678416 11015 hybrid_clock.cc:648] HybridClock initialized: now 1790593111678362 us; error 40 us; skew 500 ppm
I20260928 10:58:31.678766 11015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593111, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:31.682225 11015 init.cc:377] Logged in from keytab as oryx/127.3.252.65@KRBTEST.COM (short username oryx)
I20260928 10:58:31.683604 11015 webserver.cc:460] Webserver started at http://127.3.252.65:45997/ using document root <none> and password file <none>
I20260928 10:58:31.684262 11015 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:31.684378 11015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:31.689595 11015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:31.693033 11108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:31.694526 11015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:31.694674 11015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "75048a2576194db5ba5cfcb17b09babc"
format_stamp: "Formatted at 2026-09-28 10:58:28 on dist-test-slave-djjw"
server_key: "1f7d38f3eb274e571a0c3fac941a046b"
server_key_iv: "b7558ef8c72fa1669472848fba175556"
server_key_version: "encryptionkey@0"
I20260928 10:58:31.695659 11015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:31.712447 11015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:31.716542 11015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:31.716775 11015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:31.717685 11015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:31.719004 11015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:31.719117 11015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:31.719218 11015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:31.719285 11015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:31.751078 11015 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:42053
I20260928 10:58:31.751169 11220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:42053 every 8 connection(s)
I20260928 10:58:31.752463 11015 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:31.761148 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 11015
I20260928 10:58:31.761483 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 10450
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593111, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593111, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.125@KRBTEST.COM
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593111, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.124@KRBTEST.COM
I20260928 10:58:31.780391 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.755731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:51451 (local address 127.3.252.125:43305)
0928 10:58:31.756983 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.756990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.758264 (+ 1274us) server_negotiation.cc:407] Connection header received
0928 10:58:31.760053 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.760059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.760143 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.760222 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.763124 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.764272 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.766520 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.767389 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.767509 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.772846 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.772854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.772860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.772907 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.776498 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.777595 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.777602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.777606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.777756 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.778235 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.778240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.778243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.778585 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.778752 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.779776 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.780002 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1124,"thread_start_us":1013,"threads_started":1}
I20260928 10:58:31.780874 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.758486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:54261 (local address 127.3.252.126:33045)
0928 10:58:31.758659 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.758665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.758970 (+ 305us) server_negotiation.cc:407] Connection header received
0928 10:58:31.760053 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.760059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.760140 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.760222 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.763171 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.764241 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.766099 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.766703 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.766802 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.772353 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.772360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.772366 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.772410 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.776265 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.778792 (+ 2527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.778797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.778801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.778950 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.779405 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.779410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.779413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.779744 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.779865 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.780357 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.780578 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:58:31.782744 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.758398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.65:54261)
0928 10:58:31.758876 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.758892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.759786 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.760475 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.760484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.762001 (+ 1517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.762468 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.762498 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.764428 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.764433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.765935 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.765950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.767188 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.767201 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.767517 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.768398 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.768429 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.772067 (+ 3638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.778224 (+ 6157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.778228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.778231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.778659 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.779085 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.779089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.779092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.779275 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.779936 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.779941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.780078 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.781035 (+ 957us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.781370 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":65,"thread_start_us":265,"threads_started":1}
I20260928 10:58:31.782727 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.756580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.3.252.65:51451)
0928 10:58:31.758159 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.758173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.759786 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.760475 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.760484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.762769 (+ 2285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.762977 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.762985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.764432 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.764439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.766337 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.766353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.767956 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.767963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.768164 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.768821 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.768849 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.772586 (+ 3737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.776699 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.776711 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.776733 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.777437 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.777915 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.777920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.777924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.778104 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.778817 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.778826 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.779321 (+ 495us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.780696 (+ 1375us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.781240 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1451,"mutex_wait_us":75,"spinlock_wait_cycles":17664,"thread_start_us":1139,"threads_started":1}
I20260928 10:58:31.783281 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.755664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:59837 (local address 127.3.252.124:44755)
0928 10:58:31.755876 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.755883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.757749 (+ 1866us) server_negotiation.cc:407] Connection header received
0928 10:58:31.760040 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.760054 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.760139 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.760224 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.766119 (+ 5895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.767242 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.769809 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.770343 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.770418 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.776042 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.776049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.776054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.776097 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.779191 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.779957 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.779963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.779967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.780117 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.780541 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.780547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.780550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.780871 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.781089 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.782719 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.782972 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:58:31.784274 11222 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33045
I20260928 10:58:31.784376 11223 heartbeater.cc:347] Connected to a master server at 127.3.252.125:43305
I20260928 10:58:31.784613 11223 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:31.784621 11222 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:31.785130 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.756580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.3.252.65:59837)
0928 10:58:31.757507 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:31.757579 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:31.759765 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:31.760472 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:31.760482 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:31.761320 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:31.765936 (+ 4616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.765950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.767510 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.767515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.769619 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:31.769634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.771644 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.771648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:31.771776 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.772787 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:31.772807 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:31.775794 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:31.779349 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.779354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.779358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.779814 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.780260 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:31.780265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:31.780269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:31.780417 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:31.781190 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:31.781195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:31.781364 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 10:58:31.783937 (+ 2573us) client_negotiation.cc:240] Negotiation successful
0928 10:58:31.784242 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":343,"threads_started":1}
I20260928 10:58:31.786072 11221 heartbeater.cc:347] Connected to a master server at 127.3.252.124:44755
I20260928 10:58:31.786234 11221 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:31.786255 11222 heartbeater.cc:511] Master 127.3.252.126:33045 requested a full tablet report, sending...
I20260928 10:58:31.786432 11223 heartbeater.cc:511] Master 127.3.252.125:43305 requested a full tablet report, sending...
I20260928 10:58:31.786656 11221 heartbeater.cc:511] Master 127.3.252.124:44755 requested a full tablet report, sending...
I20260928 10:58:31.788193 10254 ts_manager.cc:194] Re-registered known tserver with Master: 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:31.788594 11043 ts_manager.cc:194] Registered new tserver with Master: 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:31.788712 10853 ts_manager.cc:194] Registered new tserver with Master: 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:31.789245 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:54261
Sep 28 10:58:31 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.66@KRBTEST.COM
I20260928 10:58:31.800244 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:31.762423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.3.252.66:39347)
0928 10:58:31.781865 (+ 19442us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:31.781874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:31.781906 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:58:31.781977 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:31.781987 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:31.782349 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:31.784405 (+ 2056us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:31.786433 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.787857 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.789671 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:31.790339 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:31.790435 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:31.794882 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:31.794892 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:31.794906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:31.794938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:31.797521 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.798304 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.798311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.798313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.798425 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:31.798748 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:31.798751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:31.798752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:31.799118 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:31.799357 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:31.799719 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 10:58:31.799908 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19219,"thread_start_us":15203,"threads_started":1}
I20260928 10:58:31.801528 10573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:31.883462 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 10450
I20260928 10:58:31.897233 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:39347
--local_ip_for_outbound_sockets=127.3.252.66
--tserver_master_addrs=127.3.252.126:33045
--webserver_port=33567
--webserver_interface=127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab
--principal=oryx/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:58:32.109453 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: NON_VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: true } } }
I20260928 10:58:32.109889 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:32.111302 11233 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: attempting to promote NON_VOTER 203df8740c5643aa8a941b846bb92d01 to VOTER
I20260928 10:58:32.111829 11233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } }
I20260928 10:58:32.113520 10863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Refusing update from remote peer 7f1cb973abd94f638b2275484463e071: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 10:58:32.113523 11053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 LEARNER]: Refusing update from remote peer 7f1cb973abd94f638b2275484463e071: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 10:58:32.114110 11233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 10:58:32.114464 11234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 10:58:32.118321 11234 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 203df8740c5643aa8a941b846bb92d01 (127.3.252.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.122534 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.122789 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:32.123087 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.123278 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:32.119035 10863 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 203df8740c5643aa8a941b846bb92d01 (127.3.252.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.126868 11053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 203df8740c5643aa8a941b846bb92d01 (127.3.252.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.129709 11234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.130038 11234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:32.134359 11238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.134783 11238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:58:32.135504 11239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1cb973abd94f638b2275484463e071" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:32.136189 11239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 10:58:32.219182 11232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:32.219470 11232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:32.219561 11232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:32.227072 11232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:32.227183 11232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:32.227308 11232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:32.237004 11232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab
--principal=oryx/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:39347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=33567
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:32.238349 11232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:32.239607 11232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:32.252238 11244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:32.252261 11245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:32.252544 11247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:32.253391 11232 server_base.cc:1034] running on GCE node
I20260928 10:58:32.254144 11232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:32.255236 11232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:32.256464 11232 hybrid_clock.cc:648] HybridClock initialized: now 1790593112256404 us; error 49 us; skew 500 ppm
I20260928 10:58:32.256845 11232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:32.260597 11232 init.cc:377] Logged in from keytab as oryx/127.3.252.66@KRBTEST.COM (short username oryx)
I20260928 10:58:32.262244 11232 webserver.cc:460] Webserver started at http://127.3.252.66:33567/ using document root <none> and password file <none>
I20260928 10:58:32.263095 11232 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:32.263243 11232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:32.268725 11232 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:58:32.271416 11254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:32.272748 11232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:58:32.273025 11232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "739dff71c550428f8a56b3d96cdb08f0"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "5143b555f216f399c8208ee2cf984b71"
server_key_iv: "31743507a26c8fa79eeb3e37f0211ca4"
server_key_version: "encryptionkey@0"
I20260928 10:58:32.274062 11232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:32.288727 11232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:32.293196 11232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:32.293464 11232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:32.294559 11232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:32.295985 11232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:32.296077 11232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:32.296172 11232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:32.296244 11232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:32.328270 11232 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:39347
I20260928 10:58:32.328341 11366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:39347 every 8 connection(s)
I20260928 10:58:32.329696 11232 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:32.339144 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 11232
I20260928 10:58:32.339453 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 10583
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.124@KRBTEST.COM
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.67@KRBTEST.COM
I20260928 10:58:32.376210 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.332466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:36797 (local address 127.3.252.126:33045)
0928 10:58:32.339489 (+ 7023us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.339495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.346803 (+ 7308us) server_negotiation.cc:407] Connection header received
0928 10:58:32.349859 (+ 3056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.349866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.349983 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.350089 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.358475 (+ 8386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.359785 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.362751 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.363371 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.363502 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.368713 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:32.368720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:32.368725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:32.368771 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:32.372275 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.373446 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.373451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.373455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.373613 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.374168 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.374173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.374177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.374512 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:32.374632 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:32.375589 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 10:58:32.375828 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6879,"thread_start_us":6745,"threads_started":1}
I20260928 10:58:32.378778 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.333064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:36797)
0928 10:58:32.340001 (+ 6937us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:32.340061 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:32.349211 (+ 9150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:32.353486 (+ 4275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:32.353501 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:32.354503 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:32.355508 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.355540 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.360160 (+ 4620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.360165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.362538 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.362571 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.363874 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.363881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.364165 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.364899 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:32.364928 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:32.368438 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:32.372513 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.372522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.372541 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.373100 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.373769 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.373774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.373777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.374042 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.374739 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:32.374747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:32.375185 (+ 438us) client_negotiation.cc:769] Sending connection context
0928 10:58:32.376821 (+ 1636us) client_negotiation.cc:240] Negotiation successful
0928 10:58:32.377358 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6625,"thread_start_us":6376,"threads_started":1}
I20260928 10:58:32.380388 11368 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33045
I20260928 10:58:32.380722 11368 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:32.381134 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.340840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.3.252.67:34109)
0928 10:58:32.358466 (+ 17626us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.358476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.359475 (+ 999us) server_negotiation.cc:407] Connection header received
0928 10:58:32.359573 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.359582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.359880 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.360057 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.362725 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.364272 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.366115 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.366989 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.367091 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.373960 (+ 6869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:32.373977 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:32.373994 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:32.374039 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:32.377627 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.378492 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.378501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.378504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.378918 (+ 414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.379341 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.379346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.379350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.379716 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:32.379979 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:32.380510 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 10:58:32.380764 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17413,"thread_start_us":17647,"threads_started":1}
I20260928 10:58:32.381656 11368 heartbeater.cc:511] Master 127.3.252.126:33045 requested a full tablet report, sending...
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.125@KRBTEST.COM
I20260928 10:58:32.383157 10706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:32.383981 10254 ts_manager.cc:194] Re-registered known tserver with Master: 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66:39347)
I20260928 10:58:32.384429 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.333064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.3.252.66:58755)
0928 10:58:32.359155 (+ 26091us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:32.359182 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:32.359322 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:32.362591 (+ 3269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:32.362596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:32.363228 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:32.363582 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.363594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.365257 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.365266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.367834 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.367850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.369081 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.369088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.369255 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.369874 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:32.369897 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:32.372310 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:32.379591 (+ 7281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.379596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.379600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.380557 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.382500 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.382504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.382508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.382643 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.383234 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:32.383239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:32.383406 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 10:58:32.383579 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 10:58:32.383946 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25925,"mutex_wait_us":36,"spinlock_wait_cycles":34176,"thread_start_us":27018,"threads_started":1}
I20260928 10:58:32.385109 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:36797
I20260928 10:58:32.385707 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.332637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:58755 (local address 127.3.252.124:44755)
0928 10:58:32.362155 (+ 29518us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.362162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.362182 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:58:32.362237 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.362242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.362317 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.362425 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.363767 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.365076 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.368004 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.368692 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.368819 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.372522 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:32.372529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:32.372533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:32.372568 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:32.379118 (+ 6550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.382223 (+ 3105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.382228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.382231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.382368 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.382771 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.382776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.382779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.383087 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:32.383230 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:32.384746 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0928 10:58:32.385108 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29392,"thread_start_us":29454,"threads_started":1}
I20260928 10:58:32.386808 11367 heartbeater.cc:347] Connected to a master server at 127.3.252.124:44755
I20260928 10:58:32.386945 11367 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:32.387295 11367 heartbeater.cc:511] Master 127.3.252.124:44755 requested a full tablet report, sending...
I20260928 10:58:32.388394 11043 ts_manager.cc:194] Registered new tserver with Master: 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66:39347)
I20260928 10:58:32.391839 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.340780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:47041 (local address 127.3.252.125:43305)
0928 10:58:32.359749 (+ 18969us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.359754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.362047 (+ 2293us) server_negotiation.cc:407] Connection header received
0928 10:58:32.362421 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.362428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.362517 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.362596 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.363973 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.365032 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.375830 (+ 10798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.376478 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.376575 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.384347 (+ 7772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:32.384355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:32.384359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:32.384415 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:32.388553 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.389418 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.389426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.389429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.389586 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.390149 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.390155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.390158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.390518 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:32.390657 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:32.391271 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 10:58:32.391524 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18859,"thread_start_us":19123,"threads_started":1}
I20260928 10:58:32.392513 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.340771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.3.252.66:47041)
0928 10:58:32.361924 (+ 21153us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:32.361950 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:32.362262 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:32.362796 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:32.362801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:32.363424 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:32.363805 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.363821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.373307 (+ 9486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.373312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.374749 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.374759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.378217 (+ 3458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.378224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.378444 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.380744 (+ 2300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:32.380765 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:32.384114 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:32.388731 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.388736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.388741 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.389273 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.389772 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.389777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.389781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.390014 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.390739 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:32.390744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:32.390924 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 10:58:32.392017 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0928 10:58:32.392257 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20996,"thread_start_us":20829,"threads_started":1}
I20260928 10:58:32.393195 11369 heartbeater.cc:347] Connected to a master server at 127.3.252.125:43305
I20260928 10:58:32.393303 11369 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:32.393610 11369 heartbeater.cc:511] Master 127.3.252.125:43305 requested a full tablet report, sending...
I20260928 10:58:32.394528 10853 ts_manager.cc:194] Registered new tserver with Master: 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66:39347)
I20260928 10:58:32.470475 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 10583
I20260928 10:58:32.484172 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:34109
--local_ip_for_outbound_sockets=127.3.252.67
--tserver_master_addrs=127.3.252.126:33045
--webserver_port=40215
--webserver_interface=127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab
--principal=oryx/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:58:32.791823 11222 heartbeater.cc:503] Master 127.3.252.126:33045 was elected leader, sending a full tablet report...
W20260928 10:58:32.805866 11380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:32.806151 11380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:32.806306 11380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:32.813437 11380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:32.813540 11380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:32.813699 11380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:32.823283 11380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab
--principal=oryx/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:34109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=40215
--tserver_master_addrs=127.3.252.126:33045,127.3.252.125:43305,127.3.252.124:44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:32.824620 11380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:32.825949 11380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:32.838321 11386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:32.838292 11385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:32.838367 11388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:32.839145 11380 server_base.cc:1034] running on GCE node
I20260928 10:58:32.839701 11380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:32.840575 11380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:32.841758 11380 hybrid_clock.cc:648] HybridClock initialized: now 1790593112841713 us; error 53 us; skew 500 ppm
I20260928 10:58:32.842049 11380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:32.844897 11380 init.cc:377] Logged in from keytab as oryx/127.3.252.67@KRBTEST.COM (short username oryx)
I20260928 10:58:32.846246 11380 webserver.cc:460] Webserver started at http://127.3.252.67:40215/ using document root <none> and password file <none>
I20260928 10:58:32.846889 11380 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:32.847007 11380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:32.852044 11380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 10:58:32.854650 11395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:32.855914 11380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 10:58:32.856055 11380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "1cc9822af1d7416dac64358e3b58a664"
format_stamp: "Formatted at 2026-09-28 10:58:29 on dist-test-slave-djjw"
server_key: "0b2e62b7409e3fe57de8e2ec704df332"
server_key_iv: "543920d2c03a84ce06da08c98083ca61"
server_key_version: "encryptionkey@0"
I20260928 10:58:32.856974 11380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:32.877804 11380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:32.881976 11380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:32.882222 11380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:32.883172 11380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:32.884524 11380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:32.884617 11380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:32.884725 11380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:32.884800 11380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:32.915307 11380 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:34109
I20260928 10:58:32.915452 11507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:34109 every 8 connection(s)
I20260928 10:58:32.916702 11380 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:32.924080 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 11380
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.125@KRBTEST.COM
Sep 28 10:58:32 dist-test-slave-djjw krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593112, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.124@KRBTEST.COM
I20260928 10:58:32.988288 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.936196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:51961 (local address 127.3.252.126:33045)
0928 10:58:32.943372 (+ 7176us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.943379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.943810 (+ 431us) server_negotiation.cc:407] Connection header received
0928 10:58:32.943936 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.943941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.944027 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.944135 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.946169 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.947197 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.954516 (+ 7319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.955268 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.955412 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.960929 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:32.960936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:32.960942 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:32.960992 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:32.978816 (+ 17824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.981399 (+ 2583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.981407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.981411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.981634 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.985945 (+ 4311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.985952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.985956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.986394 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:32.986588 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:32.987588 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0928 10:58:32.987855 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7048,"thread_start_us":6940,"threads_started":1}
I20260928 10:58:32.989301 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.936049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.67:51961)
0928 10:58:32.943687 (+ 7638us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:32.943705 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:32.943823 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:32.944601 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:32.944611 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:32.945271 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:32.945961 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.945989 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.947359 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.947363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.949394 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.949408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.955530 (+ 6122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.955538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.955909 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.956797 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:32.956831 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:32.960665 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:32.979004 (+ 18339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.979018 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.979042 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.979829 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.981751 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.981756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.981760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.982055 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.986542 (+ 4487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:32.986553 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:32.987058 (+ 505us) client_negotiation.cc:769] Sending connection context
0928 10:58:32.987328 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 10:58:32.987882 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7549,"thread_start_us":6787,"threads_started":1}
I20260928 10:58:32.994444 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.919972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.3.252.67:45611)
0928 10:58:32.942599 (+ 22627us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:32.942627 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:32.942767 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:32.947490 (+ 4723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:32.947495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:32.948206 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:32.948546 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.948558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.959084 (+ 10526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.959089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.960654 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.960664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.967225 (+ 6561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.967232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.967433 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.968121 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:32.968159 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:32.970995 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:32.983693 (+ 12698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.983699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.983703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.984257 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.989999 (+ 5742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.990005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.990010 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.990174 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.992986 (+ 2812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:32.992991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:32.993157 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 10:58:32.993274 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 10:58:32.993562 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22443,"mutex_wait_us":21,"spinlock_wait_cycles":76160,"thread_start_us":14714,"threads_started":1}
I20260928 10:58:32.995029 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.919751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:45611 (local address 127.3.252.125:43305)
0928 10:58:32.947142 (+ 27391us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.947149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.947168 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:58:32.947228 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.947242 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.947329 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.947441 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.957747 (+ 10306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.958889 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.966203 (+ 7314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.966986 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.967132 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.976539 (+ 9407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:32.976549 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:32.976554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:32.976622 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:32.983539 (+ 6917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.989665 (+ 6126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.989671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.989676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.989882 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.992433 (+ 2551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.992440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.992444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.992849 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:32.993012 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:32.993408 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 10:58:32.993630 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27265,"thread_start_us":27159,"threads_started":1}
I20260928 10:58:32.996430 11510 heartbeater.cc:347] Connected to a master server at 127.3.252.125:43305
I20260928 10:58:32.996767 11510 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:32.997624 11510 heartbeater.cc:511] Master 127.3.252.125:43305 requested a full tablet report, sending...
I20260928 10:58:32.999495 10853 ts_manager.cc:194] Registered new tserver with Master: 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109)
I20260928 10:58:33.000701 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.919972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.3.252.67:60811)
0928 10:58:32.934694 (+ 14722us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:32.934752 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:32.938753 (+ 4001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:32.947961 (+ 9208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:32.947966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:32.948432 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:32.948737 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.948749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.963334 (+ 14585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.963342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.965007 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:32.965026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.971077 (+ 6051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.971086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:32.971323 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.971995 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:32.972018 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:32.974843 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:32.993960 (+ 19117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.993966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.993971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.994492 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.997387 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:32.997394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:32.997399 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:32.998009 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:32.999622 (+ 1613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:32.999627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:32.999779 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 10:58:32.999932 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 10:58:33.000271 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14308,"thread_start_us":23807,"threads_started":1}
I20260928 10:58:33.003129 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.919432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:60811 (local address 127.3.252.124:44755)
0928 10:58:32.946496 (+ 27064us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.946502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.946533 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:58:32.946595 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.946601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.946694 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.948894 (+ 2200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.949064 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.950296 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.965349 (+ 15053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.966071 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.975093 (+ 9022us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:32.975404 (+ 311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:32.975410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:32.975416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:32.975446 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:32.993351 (+ 17905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.995063 (+ 1712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.995069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.995073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.996650 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:32.998179 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:32.998185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:32.998189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:32.998568 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.001139 (+ 2571us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.001290 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.002400 (+ 1110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26929,"thread_start_us":26813,"threads_started":1}
I20260928 10:58:33.004122 11508 heartbeater.cc:347] Connected to a master server at 127.3.252.124:44755
I20260928 10:58:33.004276 11508 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:33.004742 11508 heartbeater.cc:511] Master 127.3.252.124:44755 requested a full tablet report, sending...
I20260928 10:58:33.006354 11043 ts_manager.cc:194] Registered new tserver with Master: 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109)
I20260928 10:58:33.007717 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.950837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.3.252.124:44755)
0928 10:58:32.969610 (+ 18773us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.969615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.987270 (+ 17655us) server_negotiation.cc:407] Connection header received
0928 10:58:32.987446 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.987451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.987529 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.987627 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.989788 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.991336 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.995485 (+ 4149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.996669 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:32.996765 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.000721 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:33.000728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:33.000732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:33.000769 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:33.004347 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.005201 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.005206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.005209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.005339 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.005752 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.005758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.005761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.006162 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.006441 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.007156 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.007375 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18603,"thread_start_us":19935,"threads_started":1}
I20260928 10:58:33.009155 11509 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33045
I20260928 10:58:33.009289 11509 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:33.009721 11509 heartbeater.cc:511] Master 127.3.252.126:33045 requested a full tablet report, sending...
I20260928 10:58:33.010823 10254 ts_manager.cc:194] Re-registered known tserver with Master: 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109)
I20260928 10:58:33.011931 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:51961
I20260928 10:58:33.016433 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.949419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.3.252.126:33045)
0928 10:58:32.988548 (+ 39129us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.988554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.988573 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:58:32.988624 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.988628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.988719 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.988817 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.994669 (+ 5852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.999418 (+ 4749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.001927 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.007343 (+ 5416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.007480 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.010275 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:33.010282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:33.010286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:33.010329 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:33.013699 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.014422 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.014426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.014428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.014622 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.014995 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.014999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.015001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.015292 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.015554 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.015874 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.016082 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39014,"thread_start_us":58706,"threads_started":1}
I20260928 10:58:33.016862 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:32.950252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.3.252.125:43305)
0928 10:58:32.989350 (+ 39098us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:32.989357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:32.989377 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:58:32.989432 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:32.989438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:32.989519 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:32.993496 (+ 3977us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:32.993739 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:32.996464 (+ 2725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.003158 (+ 6694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.004074 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.004914 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.010414 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:33.010422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:33.010427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:33.010463 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:33.013335 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.014243 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.014249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.014253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.014413 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.014778 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.014783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.014787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.015095 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.015305 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.016311 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.016519 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38952,"thread_start_us":42139,"threads_started":1}
I20260928 10:58:33.020121 10254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:58:33.027339 10254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:33.058706 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.047135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.3.252.67:34109)
0928 10:58:33.047722 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.047731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.047766 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:58:33.047867 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.047875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.048139 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.048337 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:33.049749 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.051796 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.054672 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.056567 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.056711 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.056943 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.058264 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.058406 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":600,"threads_started":1}
I20260928 10:58:33.058789 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.046573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:34109 (local address 127.0.0.1:49554)
0928 10:58:33.047225 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:33.047251 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:33.047383 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:33.048336 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:33.048341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:33.048371 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:33.048943 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.048958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.051944 (+ 2986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.051950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.054492 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.054513 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.056777 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.056784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.057593 (+ 809us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.057634 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 10:58:33.058209 (+ 575us) client_negotiation.cc:240] Negotiation successful
0928 10:58:33.058387 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":1898,"threads_started":1}
I20260928 10:58:33.063552 11443 tablet_service.cc:1467] Processing CreateTablet for tablet 192e1935518b48baac0e1829454796aa (DEFAULT_TABLE table=test-table [id=a9ae7d13c85346f9ad43d4caa3a3a4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:33.065631 11443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192e1935518b48baac0e1829454796aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:33.068259 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.048824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:39347 (local address 127.0.0.1:59114)
0928 10:58:33.059024 (+ 10200us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:33.059044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:33.059176 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:33.060068 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:33.060074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:33.060102 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:33.060346 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.060359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.063116 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.063153 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.065244 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.065261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.067433 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.067439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.067696 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.067739 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:58:33.067842 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 10:58:33.067991 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10096,"mutex_wait_us":24,"thread_start_us":9210,"threads_started":1}
I20260928 10:58:33.069061 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.047085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42053 (local address 127.0.0.1:50852)
0928 10:58:33.059021 (+ 11936us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:33.059042 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:33.059265 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:33.060173 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:33.060177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:33.060202 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:33.060557 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.060569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.062554 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.062560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.065461 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.065473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.067479 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.067485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.068515 (+ 1030us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.068541 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:58:33.068625 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:58:33.068749 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11085,"mutex_wait_us":832,"thread_start_us":20654,"threads_started":1}
I20260928 10:58:33.069165 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.047622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.3.252.65:42053)
0928 10:58:33.048147 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.048156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.059154 (+ 10998us) server_negotiation.cc:407] Connection header received
0928 10:58:33.059466 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.059475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.059759 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.059928 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:33.060756 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.062352 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.065621 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.066994 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.067102 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.067298 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.068709 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.068874 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":1308,"threads_started":1}
I20260928 10:58:33.069855 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.049122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.3.252.66:39347)
0928 10:58:33.049672 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.049682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.059497 (+ 9815us) server_negotiation.cc:407] Connection header received
0928 10:58:33.059603 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.059610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.059854 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.060608 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:33.060823 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.062891 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.065417 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.066952 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.067097 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.067325 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.069384 (+ 2059us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.069557 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":327,"threads_started":1}
I20260928 10:58:33.072831 11156 tablet_service.cc:1467] Processing CreateTablet for tablet 192e1935518b48baac0e1829454796aa (DEFAULT_TABLE table=test-table [id=a9ae7d13c85346f9ad43d4caa3a3a4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:33.074097 11156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192e1935518b48baac0e1829454796aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:33.077206 11302 tablet_service.cc:1467] Processing CreateTablet for tablet 192e1935518b48baac0e1829454796aa (DEFAULT_TABLE table=test-table [id=a9ae7d13c85346f9ad43d4caa3a3a4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:33.078627 11302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192e1935518b48baac0e1829454796aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:33.079280 11542 tablet_bootstrap.cc:492] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Bootstrap starting.
I20260928 10:58:33.082562 11542 tablet_bootstrap.cc:654] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:33.083871 11542 log.cc:824] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:33.086450 11542 tablet_bootstrap.cc:492] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: No bootstrap required, opened a new log
I20260928 10:58:33.086706 11542 ts_tablet_manager.cc:1397] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260928 10:58:33.089665 11544 tablet_bootstrap.cc:492] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Bootstrap starting.
I20260928 10:58:33.094156 11544 tablet_bootstrap.cc:654] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:33.094552 11545 tablet_bootstrap.cc:492] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Bootstrap starting.
I20260928 10:58:33.095599 11544 log.cc:824] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:33.096509 11542 raft_consensus.cc:348] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.096999 11542 raft_consensus.cc:374] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:33.097098 11542 raft_consensus.cc:729] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc9822af1d7416dac64358e3b58a664, State: Initialized, Role: FOLLOWER
I20260928 10:58:33.097771 11542 consensus_queue.cc:260] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.097978 11545 tablet_bootstrap.cc:654] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:33.099196 11545 log.cc:824] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:33.099556 11542 ts_tablet_manager.cc:1428] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260928 10:58:33.099843 11509 heartbeater.cc:503] Master 127.3.252.126:33045 was elected leader, sending a full tablet report...
I20260928 10:58:33.101922 11544 tablet_bootstrap.cc:492] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: No bootstrap required, opened a new log
I20260928 10:58:33.102224 11544 ts_tablet_manager.cc:1397] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260928 10:58:33.102618 11545 tablet_bootstrap.cc:492] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: No bootstrap required, opened a new log
I20260928 10:58:33.102861 11545 ts_tablet_manager.cc:1397] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 10:58:33.111420 11545 raft_consensus.cc:348] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.111778 11545 raft_consensus.cc:374] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:33.111867 11545 raft_consensus.cc:729] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739dff71c550428f8a56b3d96cdb08f0, State: Initialized, Role: FOLLOWER
I20260928 10:58:33.112038 11544 raft_consensus.cc:348] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.112392 11544 raft_consensus.cc:374] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:33.112478 11544 raft_consensus.cc:729] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75048a2576194db5ba5cfcb17b09babc, State: Initialized, Role: FOLLOWER
I20260928 10:58:33.112411 11545 consensus_queue.cc:260] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.112987 11544 consensus_queue.cc:260] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.114537 11544 ts_tablet_manager.cc:1428] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260928 10:58:33.114534 11545 ts_tablet_manager.cc:1428] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260928 10:58:33.114778 11368 heartbeater.cc:503] Master 127.3.252.126:33045 was elected leader, sending a full tablet report...
I20260928 10:58:33.149503 11549 raft_consensus.cc:482] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:33.149816 11549 raft_consensus.cc:504] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.151525 11549 leader_election.cc:288] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109), 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:33.157917 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.151883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:60371 (local address 127.3.252.67:34109)
0928 10:58:33.152087 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.152093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.152931 (+ 838us) server_negotiation.cc:407] Connection header received
0928 10:58:33.153114 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.153118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.153182 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.153252 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:33.153698 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.154509 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.156203 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.156950 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.157183 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.157259 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.157509 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.157628 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:58:33.158195 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.151992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:58189 (local address 127.3.252.65:42053)
0928 10:58:33.152211 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.152218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.153101 (+ 883us) server_negotiation.cc:407] Connection header received
0928 10:58:33.153374 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.153380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.153450 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.153509 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:33.154254 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.154931 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.156617 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.157381 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.157520 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.157589 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.157887 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.158007 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:58:33.158252 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.151788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:34109 (local address 127.3.252.66:60371)
0928 10:58:33.152814 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:33.152833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:33.152978 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:33.153290 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:33.153295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:33.153326 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:33.153565 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.153574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.154640 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.154644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.156052 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.156063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.157186 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.157193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.157357 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.157380 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 10:58:33.157463 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 10:58:33.157598 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":3,"thread_start_us":692,"threads_started":1}
I20260928 10:58:33.158648 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.151973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42053 (local address 127.3.252.66:58189)
0928 10:58:33.153008 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:33.153025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:33.153202 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:33.153745 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:33.153749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:33.153771 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:33.154101 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.154111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.155065 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.155071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.156471 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:33.156483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.157584 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.157590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:33.157729 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.157751 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 10:58:33.157847 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 10:58:33.157972 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":37,"thread_start_us":494,"threads_started":1}
I20260928 10:58:33.159201 11463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192e1935518b48baac0e1829454796aa" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc9822af1d7416dac64358e3b58a664" is_pre_election: true
I20260928 10:58:33.159699 11463 raft_consensus.cc:2454] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739dff71c550428f8a56b3d96cdb08f0 in term 0.
I20260928 10:58:33.159965 11176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192e1935518b48baac0e1829454796aa" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75048a2576194db5ba5cfcb17b09babc" is_pre_election: true
I20260928 10:58:33.160482 11176 raft_consensus.cc:2454] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739dff71c550428f8a56b3d96cdb08f0 in term 0.
I20260928 10:58:33.160493 11255 leader_election.cc:302] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc9822af1d7416dac64358e3b58a664, 739dff71c550428f8a56b3d96cdb08f0; no voters:
I20260928 10:58:33.160964 11549 raft_consensus.cc:2781] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:33.161134 11549 raft_consensus.cc:482] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:33.161254 11549 raft_consensus.cc:3037] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:33.163592 11549 raft_consensus.cc:504] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.164175 11549 leader_election.cc:288] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109), 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:33.164620 11463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192e1935518b48baac0e1829454796aa" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc9822af1d7416dac64358e3b58a664"
I20260928 10:58:33.164685 11176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192e1935518b48baac0e1829454796aa" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75048a2576194db5ba5cfcb17b09babc"
I20260928 10:58:33.164806 11463 raft_consensus.cc:3037] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:33.164878 11176 raft_consensus.cc:3037] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:33.167044 11463 raft_consensus.cc:2454] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739dff71c550428f8a56b3d96cdb08f0 in term 1.
I20260928 10:58:33.167043 11176 raft_consensus.cc:2454] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739dff71c550428f8a56b3d96cdb08f0 in term 1.
I20260928 10:58:33.167441 11255 leader_election.cc:302] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc9822af1d7416dac64358e3b58a664, 739dff71c550428f8a56b3d96cdb08f0; no voters:
I20260928 10:58:33.167693 11549 raft_consensus.cc:2781] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:33.168071 11549 raft_consensus.cc:686] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 1 LEADER]: Becoming Leader. State: Replica: 739dff71c550428f8a56b3d96cdb08f0, State: Running, Role: LEADER
I20260928 10:58:33.168630 11549 consensus_queue.cc:237] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } }
I20260928 10:58:33.173566 10254 catalog_manager.cc:5495] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "739dff71c550428f8a56b3d96cdb08f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } health_report { overall_health: HEALTHY } } }
W20260928 10:58:33.185413 11511 tablet.cc:2369] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:58:33.219096 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.214221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.3.252.66:39347)
0928 10:58:33.214395 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.214400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.214417 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:58:33.214541 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.214545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.214626 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.214745 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:33.215027 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.215830 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.217139 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.217543 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.217637 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.218072 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:33.218605 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:33.218713 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.218772 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.218833 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:58:33.227073 11463 raft_consensus.cc:1261] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Refusing update from remote peer 739dff71c550428f8a56b3d96cdb08f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:33.227073 11176 raft_consensus.cc:1261] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Refusing update from remote peer 739dff71c550428f8a56b3d96cdb08f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:33.227965 11553 consensus_queue.cc:1035] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:33.228276 11549 consensus_queue.cc:1035] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:33.240779 11556 mvcc.cc:205] Tried to move back new op lower bound from 7334269391768735744 to 7334269391542378496. Current Snapshot: MvccSnapshot[applied={T|T < 7334269391768735744}]
I20260928 10:58:33.263692 10255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:33.263962 10255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:33.272142 10255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f1cb973abd94f638b2275484463e071: Sending DeleteTablet for 3 replicas of tablet 192e1935518b48baac0e1829454796aa
I20260928 10:58:33.273222 11156 tablet_service.cc:1514] Processing DeleteTablet for tablet 192e1935518b48baac0e1829454796aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:33 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:50852
I20260928 10:58:33.274066 11443 tablet_service.cc:1514] Processing DeleteTablet for tablet 192e1935518b48baac0e1829454796aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:33 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:49554
I20260928 10:58:33.274899 11565 tablet_replica.cc:330] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: stopping tablet replica
I20260928 10:58:33.274581 11302 tablet_service.cc:1514] Processing DeleteTablet for tablet 192e1935518b48baac0e1829454796aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:33 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:59114
I20260928 10:58:33.275380 11565 raft_consensus.cc:2229] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:33.275923 11565 raft_consensus.cc:2258] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:33.277431 11567 tablet_replica.cc:330] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: stopping tablet replica
I20260928 10:58:33.277912 11567 raft_consensus.cc:2229] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:33.278956 11567 raft_consensus.cc:2258] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:33.280056 11565 ts_tablet_manager.cc:1905] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:33.281051 11567 ts_tablet_manager.cc:1905] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:33.282907 11566 tablet_replica.cc:330] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: stopping tablet replica
I20260928 10:58:33.283483 11566 raft_consensus.cc:2229] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:33.284993 11566 raft_consensus.cc:2258] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:33.287068 11565 ts_tablet_manager.cc:1918] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:33.287120 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 10223
I20260928 10:58:33.287220 11565 log.cc:1196] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/192e1935518b48baac0e1829454796aa
I20260928 10:58:33.287731 11565 ts_tablet_manager.cc:1939] T 192e1935518b48baac0e1829454796aa P 75048a2576194db5ba5cfcb17b09babc: Deleting consensus metadata
I20260928 10:58:33.288130 10285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:33.290318 11567 ts_tablet_manager.cc:1918] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:33.290457 11567 log.cc:1196] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/192e1935518b48baac0e1829454796aa
I20260928 10:58:33.290861 11567 ts_tablet_manager.cc:1939] T 192e1935518b48baac0e1829454796aa P 739dff71c550428f8a56b3d96cdb08f0: Deleting consensus metadata
I20260928 10:58:33.291304 11566 ts_tablet_manager.cc:1905] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:33.299870 11566 ts_tablet_manager.cc:1918] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:33.300067 11566 log.cc:1196] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/192e1935518b48baac0e1829454796aa
I20260928 10:58:33.300661 11566 ts_tablet_manager.cc:1939] T 192e1935518b48baac0e1829454796aa P 1cc9822af1d7416dac64358e3b58a664: Deleting consensus metadata
I20260928 10:58:33.307950 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.278140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.3.252.125:43305)
0928 10:58:33.278339 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.278346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.286392 (+ 8046us) server_negotiation.cc:407] Connection header received
0928 10:58:33.287208 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.287214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.287301 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.287382 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:33.289785 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.290958 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.295798 (+ 4840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.296505 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.296638 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.300904 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:33.300913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:33.300918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:33.300966 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:33.304348 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.305652 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.305658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.305662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.305881 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.306311 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.306315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.306317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.306680 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.306887 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.307412 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.307655 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:58:33.323490 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.286530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.3.252.124:44755)
0928 10:58:33.286713 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.286719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.308145 (+ 21426us) server_negotiation.cc:407] Connection header received
0928 10:58:33.308388 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.308394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.308488 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.308557 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:33.310436 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.311446 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.313322 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.313975 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.314114 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.317336 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:33.317344 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:33.317349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:33.317391 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:33.320642 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.321399 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.321404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.321408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.321555 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.321950 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.321955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.321958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.322353 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.322625 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.322940 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.323166 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:58:33.417382 10239 catalog_manager.cc:4841] TS 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109): tablet 192e1935518b48baac0e1829454796aa (table test-table [id=a9ae7d13c85346f9ad43d4caa3a3a4e2]) successfully deleted
I20260928 10:58:33.417722 10239 catalog_manager.cc:4841] TS 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66:39347): tablet 192e1935518b48baac0e1829454796aa (table test-table [id=a9ae7d13c85346f9ad43d4caa3a3a4e2]) successfully deleted
I20260928 10:58:33.417917 10241 catalog_manager.cc:4841] TS 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053): tablet 192e1935518b48baac0e1829454796aa (table test-table [id=a9ae7d13c85346f9ad43d4caa3a3a4e2]) successfully deleted
I20260928 10:58:33.418393 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 10223
I20260928 10:58:33.458410 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.440489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.3.252.124:44755)
0928 10:58:33.440682 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.440688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.441650 (+ 962us) server_negotiation.cc:407] Connection header received
0928 10:58:33.441937 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.441944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.442025 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.442108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:33.444670 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.445810 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.447889 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.448561 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.448665 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.452396 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:33.452404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:33.452410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:33.452462 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:33.455511 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.456318 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.456323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.456327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.456474 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.456882 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.456887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.456891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.457248 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.457513 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.457816 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.458067 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:58:33.466495 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:33.441655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.3.252.125:43305)
0928 10:58:33.441865 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:33.441871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:33.452600 (+ 10729us) server_negotiation.cc:407] Connection header received
0928 10:58:33.452859 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:33.452865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:33.452952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:33.453020 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:33.454935 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.455931 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.457578 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:33.458161 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:33.458279 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:33.461328 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:33.461336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:33.461341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:33.461383 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:33.464009 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.464754 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.464758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.464761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.464878 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:33.465218 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:33.465221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:33.465223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:33.465468 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:33.465677 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:33.466083 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 10:58:33.466238 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:58:34.250082 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.249200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.65:50783)
0928 10:58:34.249662 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.249865 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
W20260928 10:58:34.250612 11222 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:33045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
I20260928 10:58:34.251463 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.250492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.67:45669)
0928 10:58:34.251141 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.251268 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":435,"threads_started":1}
W20260928 10:58:34.251912 11509 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:33045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
I20260928 10:58:34.259358 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.258494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:48087)
0928 10:58:34.259028 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.259164 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":243,"threads_started":1}
W20260928 10:58:34.259722 11368 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:33045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
I20260928 10:58:34.772329 11580 raft_consensus.cc:482] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f1cb973abd94f638b2275484463e071)
I20260928 10:58:34.772656 11580 raft_consensus.cc:504] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } }
I20260928 10:58:34.772904 11581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f1cb973abd94f638b2275484463e071)
I20260928 10:58:34.773146 11581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } }
I20260928 10:58:34.774109 11580 leader_election.cc:288] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f1cb973abd94f638b2275484463e071 (127.3.252.126:33045), 203df8740c5643aa8a941b846bb92d01 (127.3.252.124:44755)
I20260928 10:58:34.774399 11581 leader_election.cc:288] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f1cb973abd94f638b2275484463e071 (127.3.252.126:33045), f91d1398d5554f60a9b994ea9638d2fe (127.3.252.125:43305)
I20260928 10:58:34.775794 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.774220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48062)
0928 10:58:34.775374 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.775553 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1053,"threads_started":1}
I20260928 10:58:34.775988 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.774538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48064)
0928 10:58:34.775619 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.775765 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":851,"threads_started":1}
W20260928 10:58:34.777688 11027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
W20260928 10:58:34.777822 10837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
I20260928 10:58:34.783013 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.782512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48090)
0928 10:58:34.782692 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.782819 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 10:58:34.783083 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.782412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48074)
0928 10:58:34.782569 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.782779 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 10:58:34.783185 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.774912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:44755 (local address 127.0.0.1:44694)
0928 10:58:34.776043 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.776061 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:34.776253 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:34.776678 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:34.776681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:34.776702 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:34.776904 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.776912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.778430 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.778435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.780424 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.780437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.782206 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.782213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.782405 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.782440 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:58:34.782535 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 10:58:34.782699 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"thread_start_us":208,"threads_started":1}
W20260928 10:58:34.783408 11027 leader_election.cc:334] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f1cb973abd94f638b2275484463e071 (127.3.252.126:33045): Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
I20260928 10:58:34.783589 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.774503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.3.252.124:44755)
0928 10:58:34.774980 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:34.774986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:34.776136 (+ 1150us) server_negotiation.cc:407] Connection header received
0928 10:58:34.776388 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:34.776393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:34.776450 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:34.776498 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:34.777044 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.778299 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.780612 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.781911 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.782033 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.782137 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:34.783099 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 10:58:34.783234 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":215,"threads_started":1}
W20260928 10:58:34.783915 10837 leader_election.cc:334] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f1cb973abd94f638b2275484463e071 (127.3.252.126:33045): Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
I20260928 10:58:34.784174 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f91d1398d5554f60a9b994ea9638d2fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "203df8740c5643aa8a941b846bb92d01" is_pre_election: true
I20260928 10:58:34.784485 11053 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f91d1398d5554f60a9b994ea9638d2fe in term 1.
I20260928 10:58:34.784749 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.777394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.3.252.125:43305)
0928 10:58:34.777938 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:34.777953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:34.777971 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:58:34.778022 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:34.778027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:34.778089 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:34.778174 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:34.778827 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.780110 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.782510 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.783737 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.783845 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.783955 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:34.784339 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 10:58:34.784462 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":389,"thread_start_us":214,"threads_started":1}
I20260928 10:58:34.784744 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.775833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:43305 (local address 127.0.0.1:34632)
0928 10:58:34.776220 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.776235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:34.776330 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:34.778384 (+ 2054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:34.778388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:34.778414 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:34.778645 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.778655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.780278 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.780283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.782342 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.782355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.784002 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.784008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.784190 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.784215 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:58:34.784303 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 10:58:34.784434 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":1550,"threads_started":1}
I20260928 10:58:34.784948 10840 leader_election.cc:302] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 203df8740c5643aa8a941b846bb92d01, f91d1398d5554f60a9b994ea9638d2fe; no voters: 7f1cb973abd94f638b2275484463e071
I20260928 10:58:34.785349 11580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 10:58:34.785465 11580 raft_consensus.cc:482] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7f1cb973abd94f638b2275484463e071)
I20260928 10:58:34.785354 10863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "203df8740c5643aa8a941b846bb92d01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f91d1398d5554f60a9b994ea9638d2fe" is_pre_election: true
I20260928 10:58:34.785561 11580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:58:34.787660 11580 raft_consensus.cc:504] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } }
I20260928 10:58:34.788086 10863 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 203df8740c5643aa8a941b846bb92d01 in current term 2: Already voted for candidate f91d1398d5554f60a9b994ea9638d2fe in this term.
I20260928 10:58:34.788329 11580 leader_election.cc:288] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [CANDIDATE]: Term 2 election: Requested vote from peers 7f1cb973abd94f638b2275484463e071 (127.3.252.126:33045), 203df8740c5643aa8a941b846bb92d01 (127.3.252.124:44755)
I20260928 10:58:34.788707 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.788396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48096)
0928 10:58:34.788531 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.788595 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 10:58:34.788870 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f91d1398d5554f60a9b994ea9638d2fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "203df8740c5643aa8a941b846bb92d01"
I20260928 10:58:34.789101 11053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:58:34.789203 11027 leader_election.cc:302] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 203df8740c5643aa8a941b846bb92d01; no voters: 7f1cb973abd94f638b2275484463e071, f91d1398d5554f60a9b994ea9638d2fe
I20260928 10:58:34.789772 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.789448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48104)
0928 10:58:34.789566 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.789641 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 10:58:34.790024 10837 leader_election.cc:334] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f1cb973abd94f638b2275484463e071 (127.3.252.126:33045): Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
I20260928 10:58:34.791460 11053 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f91d1398d5554f60a9b994ea9638d2fe in term 2.
I20260928 10:58:34.791621 11581 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 10:58:34.791865 10840 leader_election.cc:302] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 203df8740c5643aa8a941b846bb92d01, f91d1398d5554f60a9b994ea9638d2fe; no voters: 7f1cb973abd94f638b2275484463e071
I20260928 10:58:34.792196 11580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:58:34.792620 11580 raft_consensus.cc:686] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [term 2 LEADER]: Becoming Leader. State: Replica: f91d1398d5554f60a9b994ea9638d2fe, State: Running, Role: LEADER
I20260928 10:58:34.792982 11580 consensus_queue.cc:237] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } }
I20260928 10:58:34.795125 11590 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f91d1398d5554f60a9b994ea9638d2fe. Latest consensus state: current_term: 2 leader_uuid: "f91d1398d5554f60a9b994ea9638d2fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:34.795591 11590 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:34.797008 11592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:34.801234 11592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:34.802165 11592 catalog_manager.cc:1259] Loaded cluster ID: b7a8d97379564ff68befb672b741c40a
I20260928 10:58:34.802243 11592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:34.803390 11592 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:34.804227 11592 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Loaded TSK: 0
I20260928 10:58:34.805078 11592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:58:34.860206 10853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:58:34.867416 10853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:34.871579 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.871167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48130)
0928 10:58:34.871355 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.871431 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 10:58:34.871699 11053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [term 2 FOLLOWER]: Refusing update from remote peer f91d1398d5554f60a9b994ea9638d2fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 10:58:34.872500 11590 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 10:58:34.872726 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.872291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48144)
0928 10:58:34.872492 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.872579 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 10:58:34.872882 10837 consensus_peers.cc:487] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe -> Peer 7f1cb973abd94f638b2275484463e071 (127.3.252.126:33045): Couldn't send request to peer 7f1cb973abd94f638b2275484463e071. Status: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 10:58:34.875439 11581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f91d1398d5554f60a9b994ea9638d2fe. Latest consensus state: current_term: 2 leader_uuid: "f91d1398d5554f60a9b994ea9638d2fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:34.875738 11581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:58:34.877122 11580 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f91d1398d5554f60a9b994ea9638d2fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:34.877468 11580 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:34.877722 11581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f91d1398d5554f60a9b994ea9638d2fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f1cb973abd94f638b2275484463e071" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33045 } } peers { permanent_uuid: "f91d1398d5554f60a9b994ea9638d2fe" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 43305 } attrs { promote: false } } peers { permanent_uuid: "203df8740c5643aa8a941b846bb92d01" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 44755 } attrs { promote: false } } }
I20260928 10:58:34.878003 11581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203df8740c5643aa8a941b846bb92d01 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:58:34.894104 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.886307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:34109 (local address 127.0.0.1:49566)
0928 10:58:34.886470 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.886490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:34.886751 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:34.887736 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:34.887742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:34.887771 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:34.888003 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.888015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.889559 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.889564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.891740 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.891757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.893242 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.893249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.893459 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.893501 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 10:58:34.893585 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:58:34.893741 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 10:58:34.895409 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.886841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:39347 (local address 127.0.0.1:59130)
0928 10:58:34.887001 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.887017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:34.887104 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:34.888044 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:34.888050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:34.888091 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:34.888424 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.888435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.890426 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.890432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.892815 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.892832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.894604 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.894611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.894810 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.894848 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:58:34.894914 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 10:58:34.895063 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 10:58:34.895547 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.886404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.3.252.67:34109)
0928 10:58:34.887406 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:34.887413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:34.887431 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:58:34.887502 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:34.887508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:34.887590 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:34.888160 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:34.888352 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.889414 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.891972 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.893068 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.894885 (+ 1817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.895017 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:34.895073 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 10:58:34.895235 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":770,"threads_started":1}
I20260928 10:58:34.896344 11443 tablet_service.cc:1467] Processing CreateTablet for tablet 8f75994653bf4116862b9be68d81c37c (DEFAULT_TABLE table=test-table [id=2f8f600beac94d58bf46012aa0c0c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:34.896582 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.887211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.3.252.66:39347)
0928 10:58:34.887683 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:34.887689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:34.887709 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:58:34.887792 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:34.887797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:34.887876 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:34.888946 (+ 1070us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:34.889143 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.890265 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.893061 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.894337 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.895934 (+ 1597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.896045 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:34.896100 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 10:58:34.896268 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":494,"threads_started":1}
I20260928 10:58:34.896809 11443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f75994653bf4116862b9be68d81c37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:34.897218 11302 tablet_service.cc:1467] Processing CreateTablet for tablet 8f75994653bf4116862b9be68d81c37c (DEFAULT_TABLE table=test-table [id=2f8f600beac94d58bf46012aa0c0c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:34.897609 11302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f75994653bf4116862b9be68d81c37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:34.903818 11603 tablet_bootstrap.cc:492] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: Bootstrap starting.
I20260928 10:58:34.905766 11603 tablet_bootstrap.cc:654] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:34.908051 11602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.887590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.3.252.65:42053)
0928 10:58:34.897916 (+ 10326us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:34.897921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:34.897938 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:58:34.898005 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:34.898008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:34.898066 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:34.898148 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:34.902598 (+ 4450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.903472 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.905802 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.907032 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.907174 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.907284 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:34.907561 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 10:58:34.907702 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10185,"thread_start_us":10636,"threads_started":1}
I20260928 10:58:34.908102 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.887371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42053 (local address 127.0.0.1:50860)
0928 10:58:34.894375 (+ 7004us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.894398 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:34.894505 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:34.902176 (+ 7671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:34.902182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:34.902242 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:34.902414 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.902426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.903658 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.903664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.905559 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:34.905575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.907210 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.907217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:34.907404 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.907429 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:58:34.907514 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:58:34.907662 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6899,"thread_start_us":16448,"threads_started":1}
I20260928 10:58:34.909109 11604 tablet_bootstrap.cc:492] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: Bootstrap starting.
I20260928 10:58:34.909334 11156 tablet_service.cc:1467] Processing CreateTablet for tablet 8f75994653bf4116862b9be68d81c37c (DEFAULT_TABLE table=test-table [id=2f8f600beac94d58bf46012aa0c0c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:34.909765 11156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f75994653bf4116862b9be68d81c37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:34.910499 11603 tablet_bootstrap.cc:492] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: No bootstrap required, opened a new log
I20260928 10:58:34.910660 11603 ts_tablet_manager.cc:1397] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 10:58:34.910786 11604 tablet_bootstrap.cc:654] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:34.911341 11603 raft_consensus.cc:348] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.911521 11603 raft_consensus.cc:374] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:34.911615 11603 raft_consensus.cc:729] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739dff71c550428f8a56b3d96cdb08f0, State: Initialized, Role: FOLLOWER
I20260928 10:58:34.911842 11603 consensus_queue.cc:260] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.912736 11603 ts_tablet_manager.cc:1428] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:34.913008 11604 tablet_bootstrap.cc:492] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: No bootstrap required, opened a new log
I20260928 10:58:34.913142 11604 ts_tablet_manager.cc:1397] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 10:58:34.916735 11604 raft_consensus.cc:348] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.917019 11604 raft_consensus.cc:374] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:34.917135 11604 raft_consensus.cc:729] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc9822af1d7416dac64358e3b58a664, State: Initialized, Role: FOLLOWER
I20260928 10:58:34.917402 11604 consensus_queue.cc:260] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.918401 11604 ts_tablet_manager.cc:1428] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 10:58:34.924371 11611 tablet_bootstrap.cc:492] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: Bootstrap starting.
I20260928 10:58:34.926301 11611 tablet_bootstrap.cc:654] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:34.927436 11607 raft_consensus.cc:482] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:34.927625 11607 raft_consensus.cc:504] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.928388 11607 leader_election.cc:288] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109), 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:34.929471 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.913272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:57597)
0928 10:58:34.929080 (+ 15808us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.929218 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15679,"thread_start_us":15545,"threads_started":1}
I20260928 10:58:34.929430 11176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f75994653bf4116862b9be68d81c37c" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75048a2576194db5ba5cfcb17b09babc" is_pre_election: true
W20260928 10:58:34.930338 11257 leader_election.cc:341] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053): Illegal state: must be running to vote when last-logged opid is not known
I20260928 10:58:34.931895 11463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f75994653bf4116862b9be68d81c37c" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc9822af1d7416dac64358e3b58a664" is_pre_election: true
I20260928 10:58:34.932185 11463 raft_consensus.cc:2454] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739dff71c550428f8a56b3d96cdb08f0 in term 0.
I20260928 10:58:34.932282 11369 heartbeater.cc:503] Master 127.3.252.125:43305 was elected leader, sending a full tablet report...
I20260928 10:58:34.932406 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.919018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.67:53727)
0928 10:58:34.931939 (+ 12921us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.932118 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12745,"threads_started":1}
I20260928 10:58:34.933653 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.933276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.67:50223)
0928 10:58:34.933444 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.933523 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 10:58:34.934201 11510 heartbeater.cc:503] Master 127.3.252.125:43305 was elected leader, sending a full tablet report...
I20260928 10:58:34.934669 11611 tablet_bootstrap.cc:492] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: No bootstrap required, opened a new log
I20260928 10:58:34.934809 11611 ts_tablet_manager.cc:1397] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 10:58:34.934309 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.932918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:43291)
0928 10:58:34.934079 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.934163 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1051}
I20260928 10:58:34.934228 11255 leader_election.cc:302] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc9822af1d7416dac64358e3b58a664, 739dff71c550428f8a56b3d96cdb08f0; no voters: 75048a2576194db5ba5cfcb17b09babc
I20260928 10:58:34.935406 11607 raft_consensus.cc:2781] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:34.935531 11607 raft_consensus.cc:482] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:34.935624 11607 raft_consensus.cc:3037] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:34.935611 11611 raft_consensus.cc:348] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.935943 11611 raft_consensus.cc:374] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:34.936056 11611 raft_consensus.cc:729] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75048a2576194db5ba5cfcb17b09babc, State: Initialized, Role: FOLLOWER
I20260928 10:58:34.936290 11611 consensus_queue.cc:260] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.938429 11607 raft_consensus.cc:504] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.939073 11607 leader_election.cc:288] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109), 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053)
I20260928 10:58:34.939489 11176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f75994653bf4116862b9be68d81c37c" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75048a2576194db5ba5cfcb17b09babc"
I20260928 10:58:34.939487 11463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f75994653bf4116862b9be68d81c37c" candidate_uuid: "739dff71c550428f8a56b3d96cdb08f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc9822af1d7416dac64358e3b58a664"
I20260928 10:58:34.939744 11463 raft_consensus.cc:3037] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:34.939776 11611 ts_tablet_manager.cc:1428] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:34.940016 11223 heartbeater.cc:503] Master 127.3.252.125:43305 was elected leader, sending a full tablet report...
I20260928 10:58:34.940058 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.937349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.65:46209)
0928 10:58:34.937884 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.938011 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":239,"threads_started":1}
I20260928 10:58:34.940014 11176 raft_consensus.cc:3037] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:34.941890 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.940584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.65:42725)
0928 10:58:34.941416 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.941580 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":295,"threads_started":1}
I20260928 10:58:34.942414 11463 raft_consensus.cc:2454] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739dff71c550428f8a56b3d96cdb08f0 in term 1.
I20260928 10:58:34.942773 11255 leader_election.cc:302] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc9822af1d7416dac64358e3b58a664, 739dff71c550428f8a56b3d96cdb08f0; no voters:
I20260928 10:58:34.942909 11176 raft_consensus.cc:2454] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739dff71c550428f8a56b3d96cdb08f0 in term 1.
I20260928 10:58:34.943035 11607 raft_consensus.cc:2781] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:34.943481 11607 raft_consensus.cc:686] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 1 LEADER]: Becoming Leader. State: Replica: 739dff71c550428f8a56b3d96cdb08f0, State: Running, Role: LEADER
I20260928 10:58:34.943785 11607 consensus_queue.cc:237] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } }
I20260928 10:58:34.944725 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.944272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:35797)
0928 10:58:34.944439 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.944563 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 10:58:34.946785 10852 catalog_manager.cc:5495] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "739dff71c550428f8a56b3d96cdb08f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "739dff71c550428f8a56b3d96cdb08f0" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 } health_report { overall_health: UNKNOWN } } }
I20260928 10:58:34.965471 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.961115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.3.252.66:39347)
0928 10:58:34.961275 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:34.961279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:34.961296 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:58:34.961351 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:34.961354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:34.961413 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:34.961485 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:34.961779 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.962533 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.963780 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:34.964213 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:34.964331 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:34.964859 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:34.964994 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:34.965082 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:34.965204 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 10:58:34.965268 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 10:58:34.968539 11463 raft_consensus.cc:1261] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Refusing update from remote peer 739dff71c550428f8a56b3d96cdb08f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:34.968600 11176 raft_consensus.cc:1261] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Refusing update from remote peer 739dff71c550428f8a56b3d96cdb08f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:34.969167 11616 consensus_queue.cc:1035] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75048a2576194db5ba5cfcb17b09babc" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:34.969452 11607 consensus_queue.cc:1035] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc9822af1d7416dac64358e3b58a664" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 34109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:34.971720 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.971167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.67:59147)
0928 10:58:34.971400 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.971497 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 10:58:34.972404 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.971619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.65:51127)
0928 10:58:34.972141 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.972238 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260928 10:58:34.978298 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.977799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.67:40763)
0928 10:58:34.978017 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.978115 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 10:58:34.979067 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.978586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:56697)
0928 10:58:34.978849 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.978929 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260928 10:58:34.986475 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.986000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.65:39829)
0928 10:58:34.986161 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.986284 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 10:58:34.987413 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.986900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.3.252.66:58591)
0928 10:58:34.987155 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:34.987238 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 10:58:34.988572 10852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:34.988876 10852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:34.994043 10852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f91d1398d5554f60a9b994ea9638d2fe: Sending DeleteTablet for 3 replicas of tablet 8f75994653bf4116862b9be68d81c37c
I20260928 10:58:34.995008 11156 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f75994653bf4116862b9be68d81c37c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:34 UTC) from {username='oryx', principal='oryx/127.3.252.125@KRBTEST.COM'} at 127.0.0.1:50860
I20260928 10:58:34.995115 11302 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f75994653bf4116862b9be68d81c37c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:34 UTC) from {username='oryx', principal='oryx/127.3.252.125@KRBTEST.COM'} at 127.0.0.1:59130
I20260928 10:58:34.995077 11155 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f75994653bf4116862b9be68d81c37c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:34 UTC) from {username='oryx', principal='oryx/127.3.252.125@KRBTEST.COM'} at 127.0.0.1:50860
I20260928 10:58:34.995689 11627 tablet_replica.cc:330] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: stopping tablet replica
I20260928 10:58:34.995841 11628 tablet_replica.cc:330] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: stopping tablet replica
I20260928 10:58:34.995870 11627 raft_consensus.cc:2229] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:34.995898 11443 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f75994653bf4116862b9be68d81c37c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:34 UTC) from {username='oryx', principal='oryx/127.3.252.125@KRBTEST.COM'} at 127.0.0.1:49566
I20260928 10:58:34.996003 11628 raft_consensus.cc:2229] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:34.996141 11627 raft_consensus.cc:2258] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:34.996371 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 11015
I20260928 10:58:34.996356 11628 raft_consensus.cc:2258] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:34.997396 11628 ts_tablet_manager.cc:1905] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:35.000511 11627 ts_tablet_manager.cc:1905] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:35.003571 11628 ts_tablet_manager.cc:1918] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:35.003728 11628 log.cc:1196] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/8f75994653bf4116862b9be68d81c37c
I20260928 10:58:35.004138 11628 ts_tablet_manager.cc:1939] T 8f75994653bf4116862b9be68d81c37c P 739dff71c550428f8a56b3d96cdb08f0: Deleting consensus metadata
I20260928 10:58:35.005241 10837 catalog_manager.cc:4841] TS 739dff71c550428f8a56b3d96cdb08f0 (127.3.252.66:39347): tablet 8f75994653bf4116862b9be68d81c37c (table test-table [id=2f8f600beac94d58bf46012aa0c0c40d]) successfully deleted
I20260928 10:58:35.007241 11629 tablet_replica.cc:330] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: stopping tablet replica
I20260928 10:58:35.007445 11629 raft_consensus.cc:2229] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:35.007673 11627 ts_tablet_manager.cc:1918] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:35.007746 11629 raft_consensus.cc:2258] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:35.007809 11627 log.cc:1196] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/8f75994653bf4116862b9be68d81c37c
I20260928 10:58:35.008231 11627 ts_tablet_manager.cc:1939] T 8f75994653bf4116862b9be68d81c37c P 75048a2576194db5ba5cfcb17b09babc: Deleting consensus metadata
I20260928 10:58:35.008939 11629 ts_tablet_manager.cc:1905] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:35.009279 10839 catalog_manager.cc:4841] TS 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053): tablet 8f75994653bf4116862b9be68d81c37c (table test-table [id=2f8f600beac94d58bf46012aa0c0c40d]) successfully deleted
W20260928 10:58:35.009729 10839 catalog_manager.cc:4808] TS 75048a2576194db5ba5cfcb17b09babc (127.3.252.65:42053): delete failed for tablet 8f75994653bf4116862b9be68d81c37c because the tablet was not found. No further retry: Not found: Tablet not found: 8f75994653bf4116862b9be68d81c37c
I20260928 10:58:35.013497 11629 ts_tablet_manager.cc:1918] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:35.013607 11629 log.cc:1196] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/8f75994653bf4116862b9be68d81c37c
I20260928 10:58:35.013980 11629 ts_tablet_manager.cc:1939] T 8f75994653bf4116862b9be68d81c37c P 1cc9822af1d7416dac64358e3b58a664: Deleting consensus metadata
I20260928 10:58:35.014830 10837 catalog_manager.cc:4841] TS 1cc9822af1d7416dac64358e3b58a664 (127.3.252.67:34109): tablet 8f75994653bf4116862b9be68d81c37c (table test-table [id=2f8f600beac94d58bf46012aa0c0c40d]) successfully deleted
I20260928 10:58:35.015676 11602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:34.997880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.3.252.65:42053)
0928 10:58:34.998058 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:34.998064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:34.998094 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:58:34.999013 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:34.999018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:34.999104 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:34.999194 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:35.003612 (+ 4418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:35.004722 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:35.006422 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:35.006865 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:35.006978 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:35.010166 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:35.010179 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:35.010194 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:35.010224 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:35.012897 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:35.013556 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:35.013564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:35.013566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:35.013725 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:35.014159 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:35.014165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:35.014169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:35.014550 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:35.014817 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:35.015141 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 10:58:35.015375 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 10:58:35.016255 11216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:35.107630 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 11015
I20260928 10:58:35.123018 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 11232
I20260928 10:58:35.141124 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:35.124239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.3.252.66:39347)
0928 10:58:35.124427 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:35.124434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:35.124533 (+ 99us) server_negotiation.cc:407] Connection header received
0928 10:58:35.124937 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:35.124943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:35.125049 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:35.125164 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:35.127041 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:35.127905 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:35.129722 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:35.130410 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:35.130534 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:35.133987 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:35.134004 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:35.134026 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:35.134065 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:35.137589 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:35.138448 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:35.138456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:35.138458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:35.138582 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:35.139012 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:35.139019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:35.139023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:35.139514 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:35.139824 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:35.140325 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 10:58:35.140584 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:58:35.141769 11362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:35.259759 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 11232
I20260928 10:58:35.278224 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 11380
I20260928 10:58:35.280408 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:35.279961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48154)
0928 10:58:35.280152 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:35.280233 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 10:58:35.281545 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:35.281130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33045 (local address 127.0.0.1:48162)
0928 10:58:35.281317 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:35.281407 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 10:58:35.295264 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:35.279026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.3.252.67:34109)
0928 10:58:35.279234 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:35.279241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:35.279265 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:58:35.279436 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:35.279441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:35.279543 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:35.279655 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:35.281287 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:35.282338 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:35.283711 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:35.284204 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:35.284317 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:35.289580 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:35.289597 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:35.289613 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:35.289662 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:35.292478 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:35.293203 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:35.293211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:35.293213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:35.293334 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:35.293691 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:35.293695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:35.293697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:35.294122 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:35.294393 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:35.294693 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 10:58:35.294913 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:58:35.295917 11503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:35.391909 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 11380
I20260928 10:58:35.407063 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 10716
I20260928 10:58:35.408288 10883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:35.528357 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 10716
W20260928 10:58:35.544667 11029 connection.cc:535] server connection from 127.0.0.1:44694 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:58:35.545006 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 10817
I20260928 10:58:35.545962 11073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:35.657040 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 10817
2026-09-28T10:58:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9010 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:35 dist-test-slave-djjw krb5kdc[11638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:35 dist-test-slave-djjw krb5kdc[11638](info): set up 2 sockets
Sep 28 10:58:35 dist-test-slave-djjw krb5kdc[11638](info): commencing operation
krb5kdc: starting...
W20260928 10:58:36.670634 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:36 dist-test-slave-djjw krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab.
Entry for principal oryx/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab.
I20260928 10:58:36.877271 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:39351
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab
--principal=oryx/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:39351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:37.201293 11651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:37.201635 11651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:37.201773 11651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:37.209091 11651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:37.209211 11651 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:58:37.209278 11651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:37.209363 11651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:37.209434 11651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:37.209527 11651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:37.219792 11651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:39351
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.126.keytab
--principal=oryx/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:39351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:37.221083 11651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:37.222396 11651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:37.233855 11657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:37.233893 11656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:37.233932 11659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:37.234826 11651 server_base.cc:1034] running on GCE node
I20260928 10:58:37.235641 11651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:37.237115 11651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:37.238344 11651 hybrid_clock.cc:648] HybridClock initialized: now 1790593117238265 us; error 48 us; skew 500 ppm
I20260928 10:58:37.238673 11651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:37 dist-test-slave-djjw krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593117, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:37.241679 11651 init.cc:377] Logged in from keytab as oryx/127.3.252.126@KRBTEST.COM (short username oryx)
I20260928 10:58:37.243144 11651 webserver.cc:460] Webserver started at http://127.3.252.126:45971/ using document root <none> and password file <none>
I20260928 10:58:37.243882 11651 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:37.243986 11651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:37.244283 11651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:37.247268 11651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "d2cb27af48cf4c91a42eb62be337aa84"
format_stamp: "Formatted at 2026-09-28 10:58:37 on dist-test-slave-djjw"
server_key: "efe98b8e3f591d64ee03329652529535"
server_key_iv: "b0a9dbecaf6a2f02532a44a511bcf63c"
server_key_version: "encryptionkey@0"
I20260928 10:58:37.248065 11651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "d2cb27af48cf4c91a42eb62be337aa84"
format_stamp: "Formatted at 2026-09-28 10:58:37 on dist-test-slave-djjw"
server_key: "efe98b8e3f591d64ee03329652529535"
server_key_iv: "b0a9dbecaf6a2f02532a44a511bcf63c"
server_key_version: "encryptionkey@0"
I20260928 10:58:37.253376 11651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:58:37.256959 11666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:37.258440 11651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:37.258613 11651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "d2cb27af48cf4c91a42eb62be337aa84"
format_stamp: "Formatted at 2026-09-28 10:58:37 on dist-test-slave-djjw"
server_key: "efe98b8e3f591d64ee03329652529535"
server_key_iv: "b0a9dbecaf6a2f02532a44a511bcf63c"
server_key_version: "encryptionkey@0"
I20260928 10:58:37.258829 11651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:37.292033 11651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:37.296473 11651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:37.296902 11651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:37.318106 11651 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:39351
I20260928 10:58:37.318148 11727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:39351 every 8 connection(s)
I20260928 10:58:37.319449 11651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:37.323280 11728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:37.328300 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 11651
I20260928 10:58:37.328516 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:37.329257 4081 external_mini_cluster.cc:1428] Setting key c5c3a1a41573374ec42918bc7878bf1f
I20260928 10:58:37.334731 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Bootstrap starting.
I20260928 10:58:37.338330 11728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:37.339578 11728 log.cc:824] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:37.343137 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: No bootstrap required, opened a new log
Sep 28 10:58:37 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:37.351042 11728 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cb27af48cf4c91a42eb62be337aa84" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39351 } }
I20260928 10:58:37.351467 11728 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:37.351588 11728 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cb27af48cf4c91a42eb62be337aa84, State: Initialized, Role: FOLLOWER
I20260928 10:58:37.352157 11728 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cb27af48cf4c91a42eb62be337aa84" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39351 } }
I20260928 10:58:37.352372 11728 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:37.352448 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:37.331418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.3.252.126:39351)
0928 10:58:37.332230 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:37.332245 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:37.332290 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:58:37.334013 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:37.334039 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:37.334463 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:37.334796 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:37.336493 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.337803 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.339547 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.340466 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.340664 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:37.345085 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:37.345093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:37.345108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:37.345159 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:37.348281 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:37.348977 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:37.348984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:37.348991 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:37.349154 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:37.349529 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:37.349532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:37.349534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:37.350177 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:37.350456 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:37.350871 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 10:58:37.351295 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":347,"threads_started":1}
I20260928 10:58:37.352519 11728 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:37.352694 11728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:37.355067 11728 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cb27af48cf4c91a42eb62be337aa84" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39351 } }
I20260928 10:58:37.355590 11728 leader_election.cc:302] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2cb27af48cf4c91a42eb62be337aa84; no voters:
I20260928 10:58:37.356113 11728 leader_election.cc:288] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:37.356232 11733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:37.357340 11733 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [term 1 LEADER]: Becoming Leader. State: Replica: d2cb27af48cf4c91a42eb62be337aa84, State: Running, Role: LEADER
I20260928 10:58:37.357923 11733 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cb27af48cf4c91a42eb62be337aa84" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39351 } }
I20260928 10:58:37.358392 11728 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:37.360607 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2cb27af48cf4c91a42eb62be337aa84. Latest consensus state: current_term: 1 leader_uuid: "d2cb27af48cf4c91a42eb62be337aa84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cb27af48cf4c91a42eb62be337aa84" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39351 } } }
I20260928 10:58:37.361037 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:37.360922 11734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2cb27af48cf4c91a42eb62be337aa84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cb27af48cf4c91a42eb62be337aa84" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39351 } } }
I20260928 10:58:37.361171 11734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:37.366880 11741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:37.371819 11741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:37.380453 11741 catalog_manager.cc:1347] Generated new cluster ID: e2244beb0b544a5aae61aa4a2fe63efc
I20260928 10:58:37.380566 11741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:37.405048 11741 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:37.406972 11741 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:37.420125 11741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Generated new TSK 0
I20260928 10:58:37.421245 11741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab.
Entry for principal oryx/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab.
I20260928 10:58:37.476864 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39351
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab
--principal=oryx/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:37.798558 11754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:37.798848 11754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:37.798969 11754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:37.806288 11754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:37.806396 11754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:37.806505 11754 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:58:37.806608 11754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:37.816473 11754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab
--principal=oryx/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:37.817905 11754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:37.819227 11754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:37.831936 11760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:37.831959 11759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:37.831976 11762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:37.832639 11754 server_base.cc:1034] running on GCE node
I20260928 10:58:37.833279 11754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:37.834216 11754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:37.835467 11754 hybrid_clock.cc:648] HybridClock initialized: now 1790593117835383 us; error 52 us; skew 500 ppm
I20260928 10:58:37.835820 11754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:37 dist-test-slave-djjw krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593117, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:37.839109 11754 init.cc:377] Logged in from keytab as oryx/127.3.252.65@KRBTEST.COM (short username oryx)
I20260928 10:58:37.840588 11754 webserver.cc:460] Webserver started at http://127.3.252.65:45351/ using document root <none> and password file <none>
I20260928 10:58:37.841394 11754 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:37.841502 11754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:37.841812 11754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:37.844785 11754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "4718b548e0b04bcbbefee76d5642cf9e"
format_stamp: "Formatted at 2026-09-28 10:58:37 on dist-test-slave-djjw"
server_key: "925c231d7ecbad993be8462e2b492384"
server_key_iv: "10e427dc6baea6a59ecd28a24c663621"
server_key_version: "encryptionkey@0"
I20260928 10:58:37.845553 11754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "4718b548e0b04bcbbefee76d5642cf9e"
format_stamp: "Formatted at 2026-09-28 10:58:37 on dist-test-slave-djjw"
server_key: "925c231d7ecbad993be8462e2b492384"
server_key_iv: "10e427dc6baea6a59ecd28a24c663621"
server_key_version: "encryptionkey@0"
I20260928 10:58:37.850921 11754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:58:37.855149 11769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:37.856721 11754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:58:37.856889 11754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "4718b548e0b04bcbbefee76d5642cf9e"
format_stamp: "Formatted at 2026-09-28 10:58:37 on dist-test-slave-djjw"
server_key: "925c231d7ecbad993be8462e2b492384"
server_key_iv: "10e427dc6baea6a59ecd28a24c663621"
server_key_version: "encryptionkey@0"
I20260928 10:58:37.857098 11754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:37.889010 11754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:37.893019 11754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:37.893271 11754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:37.896025 11754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:37.896119 11754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:37.896265 11754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:37.896320 11754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:58:37 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593117, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:37.922000 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:37.896768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:33123 (local address 127.3.252.126:39351)
0928 10:58:37.897223 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:37.897230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:37.898541 (+ 1311us) server_negotiation.cc:407] Connection header received
0928 10:58:37.900627 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:37.900633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:37.900720 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:37.900804 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:37.903323 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.904424 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.907556 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.908206 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.908352 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:37.914134 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:37.914143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:37.914149 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:37.914194 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:37.917511 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:37.918721 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:37.918728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:37.918732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:37.918889 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:37.919502 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:37.919507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:37.919511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:37.919998 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:37.920156 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:37.921525 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0928 10:58:37.921706 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260928 10:58:37.923713 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:37.897476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39351 (local address 127.3.252.65:33123)
0928 10:58:37.898339 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:37.898396 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:37.900259 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:37.901343 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:37.901351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:37.902234 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:37.903113 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:37.903138 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.904609 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.904616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:37.907374 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:37.907393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.908504 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.908511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:37.908858 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:37.909880 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:37.909925 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:37.913655 (+ 3730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:37.917690 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:37.917701 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:37.917728 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:37.918567 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:37.919042 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:37.919047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:37.919051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:37.919369 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:37.920149 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:37.920158 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:37.921004 (+ 846us) client_negotiation.cc:769] Sending connection context
0928 10:58:37.921391 (+ 387us) client_negotiation.cc:240] Negotiation successful
0928 10:58:37.922056 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":381,"threads_started":1}
I20260928 10:58:37.934612 11754 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:40451
I20260928 10:58:37.934690 11887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:40451 every 8 connection(s)
I20260928 10:58:37.935873 11754 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:37.937100 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 11754
I20260928 10:58:37.937316 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:37.938218 4081 external_mini_cluster.cc:1428] Setting key b876093754e187b311c26c04016309ae
I20260928 10:58:37.944747 11888 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39351
I20260928 10:58:37.944989 11888 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:37.946033 11888 heartbeater.cc:511] Master 127.3.252.126:39351 requested a full tablet report, sending...
I20260928 10:58:37.948392 11683 ts_manager.cc:194] Registered new tserver with Master: 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451)
I20260928 10:58:37.950428 11683 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:33123
I20260928 10:58:37.957705 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:37.945410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:36343 (local address 127.3.252.126:39351)
0928 10:58:37.945572 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:37.945576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:37.946363 (+ 787us) server_negotiation.cc:407] Connection header received
0928 10:58:37.946531 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:37.946536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:37.946610 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:37.946713 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:37.947651 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.948552 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.950385 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.950980 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.951094 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:37.952948 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:37.952953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:37.952957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:37.952986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:37.955316 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:37.955920 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:37.955925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:37.955927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:37.956042 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:37.956378 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:37.956383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:37.956386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:37.956641 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:37.956774 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:37.957161 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 10:58:37.957361 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 10:58:37.957756 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:37.945576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39351 (local address 127.3.252.65:36343)
0928 10:58:37.946271 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:37.946289 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:37.946418 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:37.946759 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:37.946763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:37.947281 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:37.947515 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:37.947524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.948687 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.948693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:37.950203 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:37.950219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:37.951173 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:37.951178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:37.951376 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:37.951996 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:37.952013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:37.952766 (+ 753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:37.955442 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:37.955448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:37.955452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:37.955814 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:37.956144 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:37.956149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:37.956152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:37.956282 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:37.956762 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:37.956767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:37.956935 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 10:58:37.957051 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 10:58:37.957349 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for oryx/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab.
Entry for principal oryx/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab.
I20260928 10:58:37.988819 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39351
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab
--principal=oryx/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:38.304776 11894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:38.305061 11894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:38.305159 11894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:38.312256 11894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:38.312357 11894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:38.312439 11894 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:58:38.312517 11894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:38.322278 11894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab
--principal=oryx/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:38.323439 11894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:38.324692 11894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:38.336895 11899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:38.336867 11900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:38.336831 11902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:38.337632 11894 server_base.cc:1034] running on GCE node
I20260928 10:58:38.338267 11894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:38.339082 11894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:38.340299 11894 hybrid_clock.cc:648] HybridClock initialized: now 1790593118340240 us; error 35 us; skew 500 ppm
I20260928 10:58:38.340569 11894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:38 dist-test-slave-djjw krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593118, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:38.343489 11894 init.cc:377] Logged in from keytab as oryx/127.3.252.66@KRBTEST.COM (short username oryx)
I20260928 10:58:38.344767 11894 webserver.cc:460] Webserver started at http://127.3.252.66:42631/ using document root <none> and password file <none>
I20260928 10:58:38.345408 11894 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:38.345499 11894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:38.345789 11894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:38.348598 11894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde"
format_stamp: "Formatted at 2026-09-28 10:58:38 on dist-test-slave-djjw"
server_key: "db68ba6d495e9a3bdb370298cb361419"
server_key_iv: "abe1d216725a66d0c5a180b8b2c57ace"
server_key_version: "encryptionkey@0"
I20260928 10:58:38.349377 11894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde"
format_stamp: "Formatted at 2026-09-28 10:58:38 on dist-test-slave-djjw"
server_key: "db68ba6d495e9a3bdb370298cb361419"
server_key_iv: "abe1d216725a66d0c5a180b8b2c57ace"
server_key_version: "encryptionkey@0"
I20260928 10:58:38.354187 11894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 10:58:38.357381 11909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:38.358757 11894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:38.358906 11894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde"
format_stamp: "Formatted at 2026-09-28 10:58:38 on dist-test-slave-djjw"
server_key: "db68ba6d495e9a3bdb370298cb361419"
server_key_iv: "abe1d216725a66d0c5a180b8b2c57ace"
server_key_version: "encryptionkey@0"
I20260928 10:58:38.359095 11894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:38.391253 11894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:38.395149 11894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:38.395399 11894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:38.397958 11894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:38.398041 11894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:38.398181 11894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:38.398239 11894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:58:38 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593118, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:38.420732 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.398633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:51501 (local address 127.3.252.126:39351)
0928 10:58:38.398809 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:38.398815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:38.400211 (+ 1396us) server_negotiation.cc:407] Connection header received
0928 10:58:38.402568 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:38.402575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:38.402666 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:38.402783 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:38.405283 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.406265 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.408571 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.409119 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.409229 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.413981 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:38.413986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:38.413990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:38.414021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:38.416898 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.418254 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.418261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.418264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.418434 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.418942 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.418947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.418951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.419239 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:38.419392 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:38.420130 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 10:58:38.420382 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:58:38.422128 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.399215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39351 (local address 127.3.252.66:51501)
0928 10:58:38.400020 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:38.400074 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:38.402183 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:38.403124 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:38.403150 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:38.404034 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:38.405047 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.405091 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.406478 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.406482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.408366 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.408409 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.409358 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.409366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.409608 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.410381 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:38.410415 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:38.413723 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:38.417052 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.417064 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.417085 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.418105 (+ 1020us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.418557 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.418566 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.418569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.418820 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.419376 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:38.419385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:38.419772 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 10:58:38.420005 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 10:58:38.420552 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":339,"threads_started":1}
I20260928 10:58:38.444784 11894 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:37967
I20260928 10:58:38.444804 12028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:37967 every 8 connection(s)
I20260928 10:58:38.445884 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.431771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39351 (local address 127.3.252.66:55925)
0928 10:58:38.432310 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:38.432332 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:38.432453 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:38.432814 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:38.432817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:38.433226 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:38.433517 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.433526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.435047 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.435053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.436803 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.436823 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.437903 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.437912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.438123 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.438843 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:38.438869 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:38.439891 (+ 1022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:38.443191 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.443197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.443202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.443811 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.444222 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.444228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.444232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.444396 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.445004 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:38.445008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:38.445162 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 10:58:38.445295 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 10:58:38.445537 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
I20260928 10:58:38.445977 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.431662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:55925 (local address 127.3.252.126:39351)
0928 10:58:38.431847 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:38.431851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:38.432414 (+ 563us) server_negotiation.cc:407] Connection header received
0928 10:58:38.432556 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:38.432562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:38.432653 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:38.432777 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:38.433651 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.434816 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.437004 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.437500 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.437625 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.440111 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:38.440119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:38.440124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:38.440166 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:38.443057 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.443928 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.443934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.443937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.444100 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.444493 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.444496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.444498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.444821 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:38.444946 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:38.445417 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 10:58:38.445639 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:58:38.446280 11894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:38.448153 12029 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39351
I20260928 10:58:38.448375 12029 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:38.449110 12029 heartbeater.cc:511] Master 127.3.252.126:39351 requested a full tablet report, sending...
I20260928 10:58:38.449469 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 11894
I20260928 10:58:38.449635 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:38.450196 4081 external_mini_cluster.cc:1428] Setting key f14290476374b011f11d28b2e11c3e33
I20260928 10:58:38.450480 11683 ts_manager.cc:194] Registered new tserver with Master: 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967)
I20260928 10:58:38.451486 11683 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:51501
WARNING: no policy specified for oryx/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab.
Entry for principal oryx/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab.
I20260928 10:58:38.496474 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39351
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab
--principal=oryx/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:38.810992 12033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:38.811247 12033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:38.811379 12033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:38.818477 12033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:38.818583 12033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:38.818696 12033 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:58:38.818809 12033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:38.828460 12033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab
--principal=oryx/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.252.126:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:38.829687 12033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:38.831028 12033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:38.843441 12039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:38.843387 12038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:38.843487 12041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:38.844247 12033 server_base.cc:1034] running on GCE node
I20260928 10:58:38.844815 12033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:38.845695 12033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:38.846903 12033 hybrid_clock.cc:648] HybridClock initialized: now 1790593118846837 us; error 46 us; skew 500 ppm
I20260928 10:58:38.847183 12033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:38 dist-test-slave-djjw krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593118, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:38.850112 12033 init.cc:377] Logged in from keytab as oryx/127.3.252.67@KRBTEST.COM (short username oryx)
I20260928 10:58:38.851478 12033 webserver.cc:460] Webserver started at http://127.3.252.67:33587/ using document root <none> and password file <none>
I20260928 10:58:38.852139 12033 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:38.852227 12033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:38.852524 12033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:38.855378 12033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "02f50e9a50cc4466a82901ba5881f005"
format_stamp: "Formatted at 2026-09-28 10:58:38 on dist-test-slave-djjw"
server_key: "bed5b5a98b87f4f4fa58e465554e6507"
server_key_iv: "2f25425698a22dab824f5bf6ccf03263"
server_key_version: "encryptionkey@0"
I20260928 10:58:38.856110 12033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "02f50e9a50cc4466a82901ba5881f005"
format_stamp: "Formatted at 2026-09-28 10:58:38 on dist-test-slave-djjw"
server_key: "bed5b5a98b87f4f4fa58e465554e6507"
server_key_iv: "2f25425698a22dab824f5bf6ccf03263"
server_key_version: "encryptionkey@0"
I20260928 10:58:38.861109 12033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:58:38.864566 12048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:38.865912 12033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:38.866034 12033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "02f50e9a50cc4466a82901ba5881f005"
format_stamp: "Formatted at 2026-09-28 10:58:38 on dist-test-slave-djjw"
server_key: "bed5b5a98b87f4f4fa58e465554e6507"
server_key_iv: "2f25425698a22dab824f5bf6ccf03263"
server_key_version: "encryptionkey@0"
I20260928 10:58:38.866240 12033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:38.901556 12033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:38.905680 12033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:38.905959 12033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:38.908576 12033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:38.908658 12033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:38.908787 12033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:38.908850 12033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:58:38 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593118, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:38.931540 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.910124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:45363 (local address 127.3.252.126:39351)
0928 10:58:38.910345 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:38.910351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:38.910902 (+ 551us) server_negotiation.cc:407] Connection header received
0928 10:58:38.913049 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:38.913055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:38.913138 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:38.913210 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:38.915505 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.916575 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.918885 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.919558 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.919709 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.924469 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:38.924474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:38.924478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:38.924513 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:38.927496 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.928507 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.928511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.928514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.928659 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.929281 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.929286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.929288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.929638 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:38.929776 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:38.930854 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0928 10:58:38.931140 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:58:38.932371 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.909617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39351 (local address 127.3.252.67:45363)
0928 10:58:38.910736 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:38.910781 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:38.912698 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:38.913705 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:38.913713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:38.914484 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:38.915333 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.915361 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.916741 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.916746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.918713 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.918731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.919783 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.919794 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.919984 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.920826 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:38.920865 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:38.924194 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:38.927682 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.927693 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.927717 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.928340 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.928790 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.928795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.928798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.929148 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.929779 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:38.929788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:38.930339 (+ 551us) client_negotiation.cc:769] Sending connection context
0928 10:58:38.930714 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 10:58:38.931138 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":652,"threads_started":1}
I20260928 10:58:38.953442 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.937524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:58093 (local address 127.3.252.126:39351)
0928 10:58:38.941033 (+ 3509us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:38.941039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:38.941060 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:58:38.941134 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:38.941140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:38.941222 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:38.941336 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:38.942482 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.943395 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.945364 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.945818 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.945960 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.948258 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:38.948263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:38.948267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:38.948296 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:38.950905 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.951746 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.951750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.951753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.951875 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:38.952237 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:38.952240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:38.952243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:38.952506 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:38.952610 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:38.953049 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 10:58:38.953211 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3354}
I20260928 10:58:38.953778 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:38.937488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39351 (local address 127.3.252.67:58093)
0928 10:58:38.938127 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:38.938145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:38.938243 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:38.941391 (+ 3148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:38.941395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:38.941857 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:38.942329 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.942341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.943624 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.943629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.945228 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:38.945243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:38.946204 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:38.946210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:38.946318 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:38.946922 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:38.946948 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:38.948027 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:38.951033 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.951040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.951044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.951608 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.951969 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:38.951975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:38.951979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:38.952138 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:38.952598 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:38.952603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:38.952806 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 10:58:38.952954 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 10:58:38.953308 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":325,"threads_started":1}
I20260928 10:58:38.954011 11888 heartbeater.cc:503] Master 127.3.252.126:39351 was elected leader, sending a full tablet report...
I20260928 10:58:38.964223 12033 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:41355
I20260928 10:58:38.964257 12167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:41355 every 8 connection(s)
I20260928 10:58:38.965385 12033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:38.966583 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 12033
I20260928 10:58:38.966743 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:38.967317 4081 external_mini_cluster.cc:1428] Setting key 94ff9f83a1addeded072ce4f7f644f2d
I20260928 10:58:38.967475 12168 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39351
I20260928 10:58:38.967664 12168 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:38.968420 12168 heartbeater.cc:511] Master 127.3.252.126:39351 requested a full tablet report, sending...
I20260928 10:58:38.969892 11682 ts_manager.cc:194] Registered new tserver with Master: 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355)
I20260928 10:58:38.970811 11682 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:45363
I20260928 10:58:38.983980 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:58:39 dist-test-slave-djjw krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:58:39 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:39 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:39.454388 12029 heartbeater.cc:503] Master 127.3.252.126:39351 was elected leader, sending a full tablet report...
I20260928 10:58:39.973076 12168 heartbeater.cc:503] Master 127.3.252.126:39351 was elected leader, sending a full tablet report...
W20260928 10:58:39.989466 12176 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.964s user 0.001s sys 0.013s
W20260928 10:58:39.990489 11778 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:39.019690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.3.252.126:39351)
0928 10:58:39.019920 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:39.019925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:39.020026 (+ 101us) server_negotiation.cc:407] Connection header received
0928 10:58:39.020186 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:39.020191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:39.020254 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:39.020339 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:39.021567 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:39.022363 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:39.023693 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:39.024085 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:39.024183 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:39.990250 (+966067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54402: BlockingRecv error: recv got EOF from 127.0.0.1:54402 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 10:58:39 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:40.007052 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:39.994215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.3.252.126:39351)
0928 10:58:39.994389 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:39.994394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:39.994868 (+ 474us) server_negotiation.cc:407] Connection header received
0928 10:58:39.994977 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:39.994981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:39.995064 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:39.995152 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:39.996339 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:39.997127 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:39.998384 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:39.998798 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:39.998893 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.002438 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:40.002444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:40.002447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:40.002483 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:40.004923 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:40.005502 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:40.005505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:40.005508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:40.005617 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:40.005945 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:40.005950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:40.005952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:40.006246 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:40.006458 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.006652 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.006826 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:58:40.010825 11682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:58:40.017438 11682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:40.051823 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.030984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:40451 (local address 127.0.0.1:55612)
0928 10:58:40.043681 (+ 12697us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.043709 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.043834 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.045099 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.045107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.045137 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:40.045542 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.045554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.046922 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.046928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.049137 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.049148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.050576 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.050582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.050768 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.050835 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.051220 (+ 385us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.051426 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12534,"thread_start_us":10856,"threads_started":1}
I20260928 10:58:40.052448 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.029083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:37967 (local address 127.0.0.1:55532)
0928 10:58:40.030167 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.030225 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.030413 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.045433 (+ 15020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.045439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.045477 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:40.045951 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.045965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.047901 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.047910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.050110 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.050125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.051721 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.051727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.051893 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.051924 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.051993 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.052110 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":14827,"threads_started":1}
I20260928 10:58:40.052541 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.031255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.3.252.65:40451)
0928 10:58:40.044410 (+ 13155us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.044420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.044451 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:58:40.044565 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.044573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.044874 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.045077 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:40.045719 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.046761 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.049281 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.050366 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.050501 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.050725 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.052081 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.052263 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12985,"thread_start_us":14019,"threads_started":1}
I20260928 10:58:40.053279 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.029279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.3.252.66:37967)
0928 10:58:40.044538 (+ 15259us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.044548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.044583 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:58:40.044696 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.044704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.044967 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.046162 (+ 1195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:40.046399 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.047753 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.050283 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.051484 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.052649 (+ 1165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.052851 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.052911 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.053073 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15068,"thread_start_us":15848,"threads_started":1}
I20260928 10:58:40.055048 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.029398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:41355 (local address 127.0.0.1:34490)
0928 10:58:40.042202 (+ 12804us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.042228 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.042419 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.048148 (+ 5729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.048152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.048176 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:40.048568 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.048582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.050319 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.050325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.052270 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.052283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.054339 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.054345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.054515 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.054547 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.054625 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.054762 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12633,"thread_start_us":770,"threads_started":1}
I20260928 10:58:40.056468 11820 tablet_service.cc:1467] Processing CreateTablet for tablet ef405812a6f1475eb3bb129f1eb0b2aa (DEFAULT_TABLE table=test-table [id=ba17dfccb4bd4e8499f7c71c07653af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:40.056604 11959 tablet_service.cc:1467] Processing CreateTablet for tablet ef405812a6f1475eb3bb129f1eb0b2aa (DEFAULT_TABLE table=test-table [id=ba17dfccb4bd4e8499f7c71c07653af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:40.057737 11959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef405812a6f1475eb3bb129f1eb0b2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.057739 11820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef405812a6f1475eb3bb129f1eb0b2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.066356 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.029990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.3.252.67:41355)
0928 10:58:40.047120 (+ 17130us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.047129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.047158 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:58:40.047257 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.047265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.047526 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.048538 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:40.048734 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.050211 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.052701 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.054137 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.054270 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.054463 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.065965 (+ 11502us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.066122 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16971,"thread_start_us":16825,"threads_started":1}
I20260928 10:58:40.069196 12193 tablet_bootstrap.cc:492] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Bootstrap starting.
I20260928 10:58:40.070288 12098 tablet_service.cc:1467] Processing CreateTablet for tablet ef405812a6f1475eb3bb129f1eb0b2aa (DEFAULT_TABLE table=test-table [id=ba17dfccb4bd4e8499f7c71c07653af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:40.071283 12194 tablet_bootstrap.cc:492] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Bootstrap starting.
I20260928 10:58:40.072284 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef405812a6f1475eb3bb129f1eb0b2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.072654 12193 tablet_bootstrap.cc:654] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.073606 12193 log.cc:824] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:40.074640 12194 tablet_bootstrap.cc:654] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.075865 12194 log.cc:824] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:40.078977 12194 tablet_bootstrap.cc:492] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: No bootstrap required, opened a new log
I20260928 10:58:40.079238 12194 ts_tablet_manager.cc:1397] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 10:58:40.079887 12193 tablet_bootstrap.cc:492] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: No bootstrap required, opened a new log
I20260928 10:58:40.080116 12193 ts_tablet_manager.cc:1397] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260928 10:58:40.086357 12197 tablet_bootstrap.cc:492] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Bootstrap starting.
I20260928 10:58:40.088014 12193 raft_consensus.cc:348] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.088444 12193 raft_consensus.cc:374] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.088327 12194 raft_consensus.cc:348] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.088583 12193 raft_consensus.cc:729] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4718b548e0b04bcbbefee76d5642cf9e, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.088694 12194 raft_consensus.cc:374] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.088789 12194 raft_consensus.cc:729] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1643b9dc5ccc4ccc8c62bed9fbb07fde, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.089185 12193 consensus_queue.cc:260] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.089308 12194 consensus_queue.cc:260] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.089927 12197 tablet_bootstrap.cc:654] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.090991 12193 ts_tablet_manager.cc:1428] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260928 10:58:40.091311 12197 log.cc:824] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:40.092760 12194 ts_tablet_manager.cc:1428] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 10:58:40.095937 12197 tablet_bootstrap.cc:492] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: No bootstrap required, opened a new log
I20260928 10:58:40.096220 12197 ts_tablet_manager.cc:1397] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 10:58:40.102986 12197 raft_consensus.cc:348] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.103317 12197 raft_consensus.cc:374] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.103389 12197 raft_consensus.cc:729] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.103883 12197 consensus_queue.cc:260] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.105350 12197 ts_tablet_manager.cc:1428] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Time spent starting tablet: real 0.009s user 0.010s sys 0.001s
W20260928 10:58:40.194861 11889 tablet.cc:2369] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:58:40.199116 12030 tablet.cc:2369] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:58:40.217873 12169 tablet.cc:2369] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:58:40.270568 12201 raft_consensus.cc:482] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:40.270917 12201 raft_consensus.cc:504] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.272655 12201 leader_election.cc:288] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967), 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451)
I20260928 10:58:40.280699 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.272980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:51861 (local address 127.3.252.66:37967)
0928 10:58:40.273210 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.273217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.273865 (+ 648us) server_negotiation.cc:407] Connection header received
0928 10:58:40.274047 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.274053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.274132 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.274234 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:40.274805 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.275776 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.278435 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.279625 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.279799 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.279872 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.280328 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.280458 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:58:40.280745 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.273222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:47443 (local address 127.3.252.65:40451)
0928 10:58:40.273416 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.273422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.274422 (+ 1000us) server_negotiation.cc:407] Connection header received
0928 10:58:40.274596 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.274600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.274653 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.274720 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:40.275162 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.276065 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.278438 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.279680 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.279810 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.279872 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.280322 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.280458 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:58:40.280813 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.272883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:37967 (local address 127.3.252.67:51861)
0928 10:58:40.273742 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.273759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.273915 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.274265 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.274270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.274296 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:40.274671 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.274680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.275914 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.275918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.278269 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.278288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.279872 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.279883 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.280086 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.280171 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.280264 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.280406 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":97,"spinlock_wait_cycles":2944,"thread_start_us":1203,"threads_started":1}
I20260928 10:58:40.281127 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.273067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:40451 (local address 127.3.252.67:47443)
0928 10:58:40.274318 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.274334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.274477 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.274756 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.274760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.274784 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:40.275027 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.275033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.276200 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.276205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.278313 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.278323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.279900 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.279905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.280087 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.280174 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.280264 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.280825 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":93,"spinlock_wait_cycles":3072,"thread_start_us":357,"threads_started":1}
I20260928 10:58:40.282135 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef405812a6f1475eb3bb129f1eb0b2aa" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" is_pre_election: true
I20260928 10:58:40.282150 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef405812a6f1475eb3bb129f1eb0b2aa" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e" is_pre_election: true
I20260928 10:58:40.282657 11979 raft_consensus.cc:2454] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 0.
I20260928 10:58:40.282658 11840 raft_consensus.cc:2454] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 0.
I20260928 10:58:40.283242 12052 leader_election.cc:302] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 1643b9dc5ccc4ccc8c62bed9fbb07fde, 4718b548e0b04bcbbefee76d5642cf9e; no voters:
I20260928 10:58:40.283588 12201 raft_consensus.cc:2781] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:40.283720 12201 raft_consensus.cc:482] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:40.283811 12201 raft_consensus.cc:3037] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:40.285945 12201 raft_consensus.cc:504] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.286513 12201 leader_election.cc:288] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 election: Requested vote from peers 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967), 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451)
I20260928 10:58:40.286922 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef405812a6f1475eb3bb129f1eb0b2aa" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde"
I20260928 10:58:40.286935 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef405812a6f1475eb3bb129f1eb0b2aa" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e"
I20260928 10:58:40.287094 11979 raft_consensus.cc:3037] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:40.287098 11840 raft_consensus.cc:3037] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:40.289287 11979 raft_consensus.cc:2454] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 1.
I20260928 10:58:40.289281 11840 raft_consensus.cc:2454] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 1.
I20260928 10:58:40.289762 12049 leader_election.cc:302] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 1643b9dc5ccc4ccc8c62bed9fbb07fde, 4718b548e0b04bcbbefee76d5642cf9e; no voters:
I20260928 10:58:40.290114 12201 raft_consensus.cc:2781] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:40.290506 12201 raft_consensus.cc:686] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 1 LEADER]: Becoming Leader. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Running, Role: LEADER
I20260928 10:58:40.291227 12201 consensus_queue.cc:237] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } }
I20260928 10:58:40.295627 11683 catalog_manager.cc:5495] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } health_report { overall_health: UNKNOWN } } }
I20260928 10:58:40.350767 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.345718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.3.252.67:41355)
0928 10:58:40.346003 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.346007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.346027 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:58:40.346240 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.346244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.346319 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.346440 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:40.346778 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.347512 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.348718 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.349128 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.349220 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.349581 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:40.350198 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:40.350305 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.350405 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.350469 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260928 10:58:40.358398 11979 raft_consensus.cc:1261] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Refusing update from remote peer 02f50e9a50cc4466a82901ba5881f005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:40.358400 11840 raft_consensus.cc:1261] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Refusing update from remote peer 02f50e9a50cc4466a82901ba5881f005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:40.359351 12204 consensus_queue.cc:1035] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:40.359738 12201 consensus_queue.cc:1035] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:40.373347 12207 mvcc.cc:205] Tried to move back new op lower bound from 7334269420978733056 to 7334269420716548096. Current Snapshot: MvccSnapshot[applied={T|T < 7334269420978733056}]
I20260928 10:58:40.375828 12209 mvcc.cc:205] Tried to move back new op lower bound from 7334269420978733056 to 7334269420716548096. Current Snapshot: MvccSnapshot[applied={T|T < 7334269420978733056}]
I20260928 10:58:40.386374 11683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:40.386620 11683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:40.389950 11683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Sending DeleteTablet for 3 replicas of tablet ef405812a6f1475eb3bb129f1eb0b2aa
I20260928 10:58:40.390898 11959 tablet_service.cc:1514] Processing DeleteTablet for tablet ef405812a6f1475eb3bb129f1eb0b2aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:40 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55532
I20260928 10:58:40.391068 12098 tablet_service.cc:1514] Processing DeleteTablet for tablet ef405812a6f1475eb3bb129f1eb0b2aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:40 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:34490
I20260928 10:58:40.391063 11820 tablet_service.cc:1514] Processing DeleteTablet for tablet ef405812a6f1475eb3bb129f1eb0b2aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:40 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55612
I20260928 10:58:40.392194 11683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:58:40.393126 11683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:40.395869 12216 tablet_replica.cc:330] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: stopping tablet replica
I20260928 10:58:40.396979 12217 tablet_replica.cc:330] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: stopping tablet replica
I20260928 10:58:40.398690 12218 tablet_replica.cc:330] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: stopping tablet replica
I20260928 10:58:40.396396 12216 raft_consensus.cc:2229] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:40.399091 12218 raft_consensus.cc:2229] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:40.399519 12216 raft_consensus.cc:2258] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:40.399492 12217 raft_consensus.cc:2229] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:40.399945 12217 raft_consensus.cc:2258] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:40.400568 12218 raft_consensus.cc:2258] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:40.401219 11959 tablet_service.cc:1467] Processing CreateTablet for tablet 23fb727963ba48fe890936997aeb993a (DEFAULT_TABLE table=test-table [id=86f625a9e64043ed9f8df30fb36b0e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:40.401242 11820 tablet_service.cc:1467] Processing CreateTablet for tablet 23fb727963ba48fe890936997aeb993a (DEFAULT_TABLE table=test-table [id=86f625a9e64043ed9f8df30fb36b0e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:40.401651 11959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fb727963ba48fe890936997aeb993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.401669 11820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fb727963ba48fe890936997aeb993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.402446 12216 ts_tablet_manager.cc:1905] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:40.407948 12194 tablet_bootstrap.cc:492] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Bootstrap starting.
I20260928 10:58:40.407994 12193 tablet_bootstrap.cc:492] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: Bootstrap starting.
I20260928 10:58:40.409245 12194 tablet_bootstrap.cc:654] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.409459 12193 tablet_bootstrap.cc:654] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.411315 12194 tablet_bootstrap.cc:492] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: No bootstrap required, opened a new log
I20260928 10:58:40.411446 12194 ts_tablet_manager.cc:1397] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 10:58:40.412235 12194 raft_consensus.cc:348] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.412405 12194 raft_consensus.cc:374] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.412523 12194 raft_consensus.cc:729] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1643b9dc5ccc4ccc8c62bed9fbb07fde, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.412714 12216 ts_tablet_manager.cc:1918] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:40.412896 12216 log.cc:1196] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/ef405812a6f1475eb3bb129f1eb0b2aa
I20260928 10:58:40.412782 12194 consensus_queue.cc:260] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.413335 12194 ts_tablet_manager.cc:1428] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:40.401422 12098 tablet_service.cc:1467] Processing CreateTablet for tablet 23fb727963ba48fe890936997aeb993a (DEFAULT_TABLE table=test-table [id=86f625a9e64043ed9f8df30fb36b0e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:40.414420 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fb727963ba48fe890936997aeb993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.415326 12216 ts_tablet_manager.cc:1939] T ef405812a6f1475eb3bb129f1eb0b2aa P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting consensus metadata
I20260928 10:58:40.418092 11670 catalog_manager.cc:4841] TS 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967): tablet ef405812a6f1475eb3bb129f1eb0b2aa (table test-table [id=ba17dfccb4bd4e8499f7c71c07653af6]) successfully deleted
I20260928 10:58:40.420686 12197 tablet_bootstrap.cc:492] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: Bootstrap starting.
I20260928 10:58:40.422526 12197 tablet_bootstrap.cc:654] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.422999 12193 tablet_bootstrap.cc:492] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: No bootstrap required, opened a new log
I20260928 10:58:40.423110 12193 ts_tablet_manager.cc:1397] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.003s
I20260928 10:58:40.423933 12193 raft_consensus.cc:348] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.424247 12193 raft_consensus.cc:374] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.424402 12193 raft_consensus.cc:729] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4718b548e0b04bcbbefee76d5642cf9e, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.424696 12193 consensus_queue.cc:260] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.425163 12217 ts_tablet_manager.cc:1905] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:40.425334 12193 ts_tablet_manager.cc:1428] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 10:58:40.424876 12197 tablet_bootstrap.cc:492] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: No bootstrap required, opened a new log
I20260928 10:58:40.430117 12197 ts_tablet_manager.cc:1397] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 10:58:40.430941 12197 raft_consensus.cc:348] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.431115 12217 ts_tablet_manager.cc:1918] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:40.431170 12197 raft_consensus.cc:374] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.431257 12217 log.cc:1196] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/ef405812a6f1475eb3bb129f1eb0b2aa
I20260928 10:58:40.431270 12197 raft_consensus.cc:729] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.431808 12217 ts_tablet_manager.cc:1939] T ef405812a6f1475eb3bb129f1eb0b2aa P 4718b548e0b04bcbbefee76d5642cf9e: Deleting consensus metadata
I20260928 10:58:40.431747 12197 consensus_queue.cc:260] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.432341 12197 ts_tablet_manager.cc:1428] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:40.426857 12218 ts_tablet_manager.cc:1905] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:40.433614 11667 catalog_manager.cc:4841] TS 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451): tablet ef405812a6f1475eb3bb129f1eb0b2aa (table test-table [id=ba17dfccb4bd4e8499f7c71c07653af6]) successfully deleted
I20260928 10:58:40.438472 12218 ts_tablet_manager.cc:1918] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:40.438616 12218 log.cc:1196] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/ef405812a6f1475eb3bb129f1eb0b2aa
I20260928 10:58:40.439033 12218 ts_tablet_manager.cc:1939] T ef405812a6f1475eb3bb129f1eb0b2aa P 02f50e9a50cc4466a82901ba5881f005: Deleting consensus metadata
I20260928 10:58:40.440440 11667 catalog_manager.cc:4841] TS 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355): tablet ef405812a6f1475eb3bb129f1eb0b2aa (table test-table [id=ba17dfccb4bd4e8499f7c71c07653af6]) successfully deleted
I20260928 10:58:40.663575 12199 raft_consensus.cc:482] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:40.663852 12199 raft_consensus.cc:504] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.665881 12199 leader_election.cc:288] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355)
I20260928 10:58:40.672598 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.666633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:53379 (local address 127.3.252.65:40451)
0928 10:58:40.666812 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.666818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.666842 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:58:40.667171 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.667178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.667303 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.667372 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:40.668204 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.669101 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.670913 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.671722 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.671881 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.671962 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.672224 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.672340 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:58:40.672649 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.666244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:40451 (local address 127.3.252.66:53379)
0928 10:58:40.666768 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.666786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.666972 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.667578 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.667582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.667615 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:40.668017 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.668031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.669217 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.669222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.670767 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.670783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.671931 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.671937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.672079 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.672103 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.672190 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.672322 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":250,"threads_started":1}
I20260928 10:58:40.673525 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fb727963ba48fe890936997aeb993a" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e" is_pre_election: true
I20260928 10:58:40.673743 11840 raft_consensus.cc:2454] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 0.
I20260928 10:58:40.674170 11910 leader_election.cc:302] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1643b9dc5ccc4ccc8c62bed9fbb07fde, 4718b548e0b04bcbbefee76d5642cf9e; no voters:
I20260928 10:58:40.674557 12199 raft_consensus.cc:2781] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:40.674674 12199 raft_consensus.cc:482] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:40.674770 12199 raft_consensus.cc:3037] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:40.675242 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.667300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:57085 (local address 127.3.252.67:41355)
0928 10:58:40.667494 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.667500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.668246 (+ 746us) server_negotiation.cc:407] Connection header received
0928 10:58:40.668428 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.668434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.668510 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.668603 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:40.669207 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.670388 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.672740 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.673964 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.674094 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.674163 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.674828 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.674961 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:58:40.675302 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.666943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:41355 (local address 127.3.252.66:57085)
0928 10:58:40.668124 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.668147 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.668299 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.668632 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.668636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.668663 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:40.669025 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.669038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.670568 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.670572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.672561 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.672575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.674369 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.674375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.674573 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.674605 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.674747 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.674878 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":1,"thread_start_us":747,"threads_started":1}
I20260928 10:58:40.676257 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fb727963ba48fe890936997aeb993a" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f50e9a50cc4466a82901ba5881f005" is_pre_election: true
I20260928 10:58:40.676738 12122 raft_consensus.cc:2454] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 0.
I20260928 10:58:40.677294 12199 raft_consensus.cc:504] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.677814 12199 leader_election.cc:288] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 election: Requested vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355)
I20260928 10:58:40.678294 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fb727963ba48fe890936997aeb993a" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e"
I20260928 10:58:40.678349 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fb727963ba48fe890936997aeb993a" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f50e9a50cc4466a82901ba5881f005"
I20260928 10:58:40.678504 11840 raft_consensus.cc:3037] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:40.678529 12122 raft_consensus.cc:3037] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:40.681324 11840 raft_consensus.cc:2454] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 1.
I20260928 10:58:40.681676 11910 leader_election.cc:302] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1643b9dc5ccc4ccc8c62bed9fbb07fde, 4718b548e0b04bcbbefee76d5642cf9e; no voters:
I20260928 10:58:40.681726 12122 raft_consensus.cc:2454] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 1.
I20260928 10:58:40.682008 12199 raft_consensus.cc:2781] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:40.682442 12199 raft_consensus.cc:686] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 LEADER]: Becoming Leader. State: Replica: 1643b9dc5ccc4ccc8c62bed9fbb07fde, State: Running, Role: LEADER
I20260928 10:58:40.683077 12199 consensus_queue.cc:237] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.686581 11683 catalog_manager.cc:5495] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde reported cstate change: term changed from 0 to 1, leader changed from <none> to 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66). New cstate: current_term: 1 leader_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } health_report { overall_health: UNKNOWN } } }
Sep 28 10:58:40 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593117, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.126@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:40.742944 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.724669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39351 (local address 127.0.0.1:54426)
0928 10:58:40.725294 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:40.725312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:40.725425 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:40.726120 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:40.726125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:40.726780 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:40.726998 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.727011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.728097 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.728103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.729624 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:40.729641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.730420 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.730427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:40.730601 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.734924 (+ 4323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:40.734952 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:40.737660 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:40.740561 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:40.740574 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:40.740583 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:40.740969 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:40.741318 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:40.741323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:40.741326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:40.741441 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:40.741937 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:40.741947 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:40.742105 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 10:58:40.742244 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 10:58:40.742520 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":338,"threads_started":1}
I20260928 10:58:40.743002 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:40.724684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.3.252.126:39351)
0928 10:58:40.725859 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:40.725864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:40.725887 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:58:40.725933 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:40.725937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:40.725988 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:40.727106 (+ 1118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:40.727226 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.727959 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.729788 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:40.730245 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:40.730345 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:40.737894 (+ 7549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:40.737899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:40.737902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:40.737932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:40.740414 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:40.741079 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:40.741084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:40.741086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:40.741193 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:40.741534 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:40.741539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:40.741542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:40.741791 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:40.741966 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:40.742357 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 10:58:40.742645 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":1036,"thread_start_us":876,"threads_started":1}
I20260928 10:58:40.748119 11683 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:54426:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:58:40.749105 11683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:40.755985 12098 tablet_service.cc:1467] Processing CreateTablet for tablet e088fb31cf5f4719ad8721df3432c20a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cb6eeebcef04a449cfc762c97734129]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:58:40.755959 11820 tablet_service.cc:1467] Processing CreateTablet for tablet e088fb31cf5f4719ad8721df3432c20a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cb6eeebcef04a449cfc762c97734129]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:58:40.756524 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e088fb31cf5f4719ad8721df3432c20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.756526 11820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e088fb31cf5f4719ad8721df3432c20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.760787 11959 tablet_service.cc:1467] Processing CreateTablet for tablet e088fb31cf5f4719ad8721df3432c20a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cb6eeebcef04a449cfc762c97734129]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:58:40.761262 11959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e088fb31cf5f4719ad8721df3432c20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:40.763717 12193 tablet_bootstrap.cc:492] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: Bootstrap starting.
I20260928 10:58:40.765350 12193 tablet_bootstrap.cc:654] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.768389 12197 tablet_bootstrap.cc:492] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: Bootstrap starting.
I20260928 10:58:40.768683 12194 tablet_bootstrap.cc:492] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Bootstrap starting.
I20260928 10:58:40.770201 12197 tablet_bootstrap.cc:654] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.770328 12194 tablet_bootstrap.cc:654] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:40.771286 12193 tablet_bootstrap.cc:492] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: No bootstrap required, opened a new log
I20260928 10:58:40.771436 12193 ts_tablet_manager.cc:1397] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260928 10:58:40.772081 12193 raft_consensus.cc:348] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.772336 12193 raft_consensus.cc:374] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.772411 12193 raft_consensus.cc:729] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4718b548e0b04bcbbefee76d5642cf9e, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.772634 12193 consensus_queue.cc:260] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.773319 12198 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:40.773630 12198 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:58:40.776237 12193 ts_tablet_manager.cc:1428] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260928 10:58:40.780253 12194 tablet_bootstrap.cc:492] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: No bootstrap required, opened a new log
I20260928 10:58:40.780424 12194 ts_tablet_manager.cc:1397] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 10:58:40.780988 12197 tablet_bootstrap.cc:492] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: No bootstrap required, opened a new log
I20260928 10:58:40.780972 12194 raft_consensus.cc:348] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.781193 12197 ts_tablet_manager.cc:1397] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260928 10:58:40.781214 12194 raft_consensus.cc:374] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.781320 12194 raft_consensus.cc:729] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1643b9dc5ccc4ccc8c62bed9fbb07fde, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.781632 12194 consensus_queue.cc:260] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.782033 12199 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:40.782142 12197 raft_consensus.cc:348] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.782290 12199 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:58:40.782373 12197 raft_consensus.cc:374] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:40.782450 12197 raft_consensus.cc:729] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Initialized, Role: FOLLOWER
I20260928 10:58:40.782721 12194 ts_tablet_manager.cc:1428] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:40.782666 12197 consensus_queue.cc:260] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:40.783196 12197 ts_tablet_manager.cc:1428] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 10:58:40.783116 12201 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:40.783397 12201 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:58:41.185707 12201 raft_consensus.cc:482] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:41.185945 12201 raft_consensus.cc:504] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.186563 12201 leader_election.cc:288] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967)
I20260928 10:58:41.187160 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e088fb31cf5f4719ad8721df3432c20a" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" is_pre_election: true
I20260928 10:58:41.187160 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e088fb31cf5f4719ad8721df3432c20a" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e" is_pre_election: true
I20260928 10:58:41.187404 11979 raft_consensus.cc:2454] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 0.
I20260928 10:58:41.187412 11840 raft_consensus.cc:2454] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 0.
I20260928 10:58:41.187852 12049 leader_election.cc:302] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 4718b548e0b04bcbbefee76d5642cf9e; no voters:
I20260928 10:58:41.188097 12201 raft_consensus.cc:2781] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:41.188218 12201 raft_consensus.cc:482] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:41.188292 12201 raft_consensus.cc:3037] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:41.190229 12201 raft_consensus.cc:504] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.190737 12201 leader_election.cc:288] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 election: Requested vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967)
I20260928 10:58:41.191229 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e088fb31cf5f4719ad8721df3432c20a" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e"
I20260928 10:58:41.191296 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e088fb31cf5f4719ad8721df3432c20a" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde"
I20260928 10:58:41.191455 11840 raft_consensus.cc:3037] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:41.191598 11979 raft_consensus.cc:3037] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:41.193429 11840 raft_consensus.cc:2454] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 1.
I20260928 10:58:41.193518 11979 raft_consensus.cc:2454] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 1.
I20260928 10:58:41.193926 12049 leader_election.cc:302] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 4718b548e0b04bcbbefee76d5642cf9e; no voters:
I20260928 10:58:41.194216 12201 raft_consensus.cc:2781] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:41.194546 12201 raft_consensus.cc:686] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [term 1 LEADER]: Becoming Leader. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Running, Role: LEADER
I20260928 10:58:41.194816 12201 consensus_queue.cc:237] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.196280 12236 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: TxnStatusTablet state changed. Reason: New leader 02f50e9a50cc4466a82901ba5881f005. Latest consensus state: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:41.196604 12236 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:58:41.197206 12238 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:58:41.197420 12238 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:58:41.197916 11681 catalog_manager.cc:5495] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } health_report { overall_health: HEALTHY } } }
I20260928 10:58:41.224043 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.218916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:41355 (local address 127.0.0.1:34514)
0928 10:58:41.219162 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:41.219183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:41.219323 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:41.219889 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:41.219895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:41.219907 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:58:41.220079 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.220086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.221172 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.221194 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.222196 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.222207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.222918 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.222924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.223064 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.223119 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.223484 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.223505 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 10:58:41.223569 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 10:58:41.223736 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 10:58:41.224115 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.218937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.3.252.67:41355)
0928 10:58:41.219582 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.219586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.219626 (+ 40us) server_negotiation.cc:407] Connection header received
0928 10:58:41.219689 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.219695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.219756 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.219847 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:41.220247 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.221003 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.222320 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.222769 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.222888 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.223288 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.223388 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.223471 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.223667 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.223738 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":371,"threads_started":1}
I20260928 10:58:41.227907 11840 raft_consensus.cc:1261] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Refusing update from remote peer 02f50e9a50cc4466a82901ba5881f005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:41.227957 11979 raft_consensus.cc:1261] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Refusing update from remote peer 02f50e9a50cc4466a82901ba5881f005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:41.228492 12201 consensus_queue.cc:1035] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:41.228957 12236 consensus_queue.cc:1035] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:41.232018 12199 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: TxnStatusTablet state changed. Reason: New leader 02f50e9a50cc4466a82901ba5881f005. Latest consensus state: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:41.232262 12199 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:58:41.240206 12199 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:41.240482 12199 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:58:41.241642 12198 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: TxnStatusTablet state changed. Reason: New leader 02f50e9a50cc4466a82901ba5881f005. Latest consensus state: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:41.241940 12198 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:58:41.244987 12201 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:41.245296 12201 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:58:41.247032 12249 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:41.247422 12249 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 02f50e9a50cc4466a82901ba5881f005: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:58:41.249588 12198 tablet_replica.cc:439] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } } }
I20260928 10:58:41.249871 12198 tablet_replica.cc:442] T e088fb31cf5f4719ad8721df3432c20a P 4718b548e0b04bcbbefee76d5642cf9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:58:41.261059 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.248624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.3.252.66:37967)
0928 10:58:41.254347 (+ 5723us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.254354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.254936 (+ 582us) server_negotiation.cc:407] Connection header received
0928 10:58:41.255152 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.255156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.255264 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.255413 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:41.255919 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.256645 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.258591 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.259071 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.259182 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.259658 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.260290 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.260402 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.260559 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.260642 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5572,"thread_start_us":5456,"threads_started":1}
I20260928 10:58:41.263528 12199 consensus_queue.cc:1035] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:41.270932 12199 consensus_queue.cc:1035] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [LEADER]: Connected to new peer: Peer: permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:41.283191 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.275897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:39435 (local address 127.3.252.67:41355)
0928 10:58:41.276064 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.276070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.277736 (+ 1666us) server_negotiation.cc:407] Connection header received
0928 10:58:41.277908 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.277932 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.277997 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.278067 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:41.278592 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.279514 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.281463 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.281956 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.282074 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.282522 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.282684 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.282773 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.282894 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.282973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:58:41.283337 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.275900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:41355 (local address 127.3.252.66:39435)
0928 10:58:41.277596 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:41.277616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:41.277768 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:41.278104 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:41.278110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:41.278123 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:58:41.278422 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.278433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.279641 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.279647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.281316 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.281331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.282095 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.282102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.282271 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.282352 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.282780 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.282791 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:58:41.282866 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 10:58:41.283001 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"thread_start_us":1379,"threads_started":1}
I20260928 10:58:41.337742 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.322365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:57501 (local address 127.3.252.66:37967)
0928 10:58:41.322521 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.322527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.330646 (+ 8119us) server_negotiation.cc:407] Connection header received
0928 10:58:41.330925 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.330931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.331015 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.331073 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:41.332149 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.333266 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.335474 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.336085 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.336197 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.336986 (+ 789us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.337115 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.337190 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.337391 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.337477 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 10:58:41.338560 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.322051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:37967 (local address 127.3.252.67:57501)
0928 10:58:41.330538 (+ 8487us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:41.330560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:41.330759 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:41.331620 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:41.331624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:41.331638 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:58:41.331934 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.331947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.333534 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.333539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.335332 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.335347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.336476 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.336482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.336687 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.336762 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.337266 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.337276 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 10:58:41.338005 (+ 729us) client_negotiation.cc:240] Negotiation successful
0928 10:58:41.338164 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8098,"thread_start_us":7940,"threads_started":1}
I20260928 10:58:41.372989 11681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:41.373261 11681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:41.376720 11681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Sending DeleteTablet for 3 replicas of tablet 23fb727963ba48fe890936997aeb993a
I20260928 10:58:41.377604 11820 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fb727963ba48fe890936997aeb993a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:41 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55612
I20260928 10:58:41.377852 11959 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fb727963ba48fe890936997aeb993a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:41 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55532
I20260928 10:58:41.378626 12098 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fb727963ba48fe890936997aeb993a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:41 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:34490
I20260928 10:58:41.400072 12268 tablet_replica.cc:330] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: stopping tablet replica
I20260928 10:58:41.401777 12268 raft_consensus.cc:2229] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:41.402233 12268 raft_consensus.cc:2258] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:41.402701 12267 tablet_replica.cc:330] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: stopping tablet replica
I20260928 10:58:41.403957 12268 ts_tablet_manager.cc:1905] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:41.404232 12267 raft_consensus.cc:2229] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:41.404541 12267 raft_consensus.cc:2258] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:41.405709 12267 ts_tablet_manager.cc:1905] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:41.416558 12267 ts_tablet_manager.cc:1918] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:58:41.418299 12266 tablet_replica.cc:330] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: stopping tablet replica
I20260928 10:58:41.418565 12266 raft_consensus.cc:2229] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:41.419234 12266 raft_consensus.cc:2258] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:41.421684 12268 ts_tablet_manager.cc:1918] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:58:41.422271 12267 log.cc:1196] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/23fb727963ba48fe890936997aeb993a
I20260928 10:58:41.422788 12267 ts_tablet_manager.cc:1939] T 23fb727963ba48fe890936997aeb993a P 4718b548e0b04bcbbefee76d5642cf9e: Deleting consensus metadata
I20260928 10:58:41.423765 12266 ts_tablet_manager.cc:1905] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:41.424085 11667 catalog_manager.cc:4841] TS 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451): tablet 23fb727963ba48fe890936997aeb993a (table test-table [id=86f625a9e64043ed9f8df30fb36b0e3a]) successfully deleted
I20260928 10:58:41.426571 12268 log.cc:1196] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/23fb727963ba48fe890936997aeb993a
I20260928 10:58:41.427161 12268 ts_tablet_manager.cc:1939] T 23fb727963ba48fe890936997aeb993a P 02f50e9a50cc4466a82901ba5881f005: Deleting consensus metadata
I20260928 10:58:41.433315 11667 catalog_manager.cc:4841] TS 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355): tablet 23fb727963ba48fe890936997aeb993a (table test-table [id=86f625a9e64043ed9f8df30fb36b0e3a]) successfully deleted
I20260928 10:58:41.433885 12266 ts_tablet_manager.cc:1918] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:58:41.434084 12266 log.cc:1196] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/23fb727963ba48fe890936997aeb993a
I20260928 10:58:41.434628 12266 ts_tablet_manager.cc:1939] T 23fb727963ba48fe890936997aeb993a P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting consensus metadata
I20260928 10:58:41.435819 11670 catalog_manager.cc:4841] TS 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967): tablet 23fb727963ba48fe890936997aeb993a (table test-table [id=86f625a9e64043ed9f8df30fb36b0e3a]) successfully deleted
Sep 28 10:58:41 dist-test-slave-djjw krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:58:41 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:41 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:58:41.497774 12275 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:41.483378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.3.252.126:39351)
0928 10:58:41.483999 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.484005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.484642 (+ 637us) server_negotiation.cc:407] Connection header received
0928 10:58:41.484833 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.484839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.484924 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.485046 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:41.486825 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.488039 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.490031 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.490725 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.490869 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.497471 (+ 6602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54436: BlockingRecv error: recv got EOF from 127.0.0.1:54436 (error 108)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":292,"threads_started":1}
Sep 28 10:58:41 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM
I20260928 10:58:41.523567 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.500438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.3.252.126:39351)
0928 10:58:41.500602 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.500608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.506374 (+ 5766us) server_negotiation.cc:407] Connection header received
0928 10:58:41.506545 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.506551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.506638 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.506756 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:41.508284 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.509409 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.511314 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.511966 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.512107 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.516997 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:41.517004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:41.517009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:41.517054 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:41.520505 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:41.521312 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:41.521318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:41.521321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:41.521476 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:41.522081 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:41.522086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:41.522089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:41.522432 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:41.522682 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.522978 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.523206 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:58:41.527865 11681 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:58:41.528568 11681 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:41.536746 12098 tablet_service.cc:1467] Processing CreateTablet for tablet f8fb5e6e3e5d436d90543a9ee556ee38 (DEFAULT_TABLE table=test-table [id=8686ef54960c48dca683e1d906d76209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:41.536736 11959 tablet_service.cc:1467] Processing CreateTablet for tablet f8fb5e6e3e5d436d90543a9ee556ee38 (DEFAULT_TABLE table=test-table [id=8686ef54960c48dca683e1d906d76209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:41.537202 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fb5e6e3e5d436d90543a9ee556ee38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:41.537200 11959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fb5e6e3e5d436d90543a9ee556ee38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:41.543978 12282 tablet_bootstrap.cc:492] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Bootstrap starting.
I20260928 10:58:41.543931 11820 tablet_service.cc:1467] Processing CreateTablet for tablet f8fb5e6e3e5d436d90543a9ee556ee38 (DEFAULT_TABLE table=test-table [id=8686ef54960c48dca683e1d906d76209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:41.544370 11820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fb5e6e3e5d436d90543a9ee556ee38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:41.546308 12282 tablet_bootstrap.cc:654] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:41.554888 12282 tablet_bootstrap.cc:492] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: No bootstrap required, opened a new log
I20260928 10:58:41.555135 12282 ts_tablet_manager.cc:1397] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260928 10:58:41.556031 12282 raft_consensus.cc:348] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.556406 12282 raft_consensus.cc:374] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:41.556548 12282 raft_consensus.cc:729] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1643b9dc5ccc4ccc8c62bed9fbb07fde, State: Initialized, Role: FOLLOWER
I20260928 10:58:41.556718 12285 tablet_bootstrap.cc:492] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: Bootstrap starting.
I20260928 10:58:41.556929 12282 consensus_queue.cc:260] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.557631 12282 ts_tablet_manager.cc:1428] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 10:58:41.559624 12285 tablet_bootstrap.cc:654] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:41.560242 12283 tablet_bootstrap.cc:492] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: Bootstrap starting.
I20260928 10:58:41.563436 12283 tablet_bootstrap.cc:654] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:41.564307 12285 tablet_bootstrap.cc:492] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: No bootstrap required, opened a new log
I20260928 10:58:41.564455 12285 ts_tablet_manager.cc:1397] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260928 10:58:41.565189 12285 raft_consensus.cc:348] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.566589 12285 raft_consensus.cc:374] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:41.566699 12285 raft_consensus.cc:729] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4718b548e0b04bcbbefee76d5642cf9e, State: Initialized, Role: FOLLOWER
I20260928 10:58:41.566967 12285 consensus_queue.cc:260] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.570870 12285 ts_tablet_manager.cc:1428] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 10:58:41.574689 12283 tablet_bootstrap.cc:492] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: No bootstrap required, opened a new log
I20260928 10:58:41.574934 12283 ts_tablet_manager.cc:1397] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260928 10:58:41.575879 12283 raft_consensus.cc:348] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.576138 12283 raft_consensus.cc:374] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:41.576246 12283 raft_consensus.cc:729] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Initialized, Role: FOLLOWER
I20260928 10:58:41.576536 12283 consensus_queue.cc:260] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.577157 12283 ts_tablet_manager.cc:1428] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 10:58:41.701448 12198 raft_consensus.cc:482] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:41.701694 12198 raft_consensus.cc:504] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.701985 12260 raft_consensus.cc:482] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:41.702169 12260 raft_consensus.cc:504] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.702939 12260 leader_election.cc:288] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355)
I20260928 10:58:41.703263 12198 leader_election.cc:288] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967), 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355)
I20260928 10:58:41.703581 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fb5e6e3e5d436d90543a9ee556ee38" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f50e9a50cc4466a82901ba5881f005" is_pre_election: true
I20260928 10:58:41.703866 12122 raft_consensus.cc:2454] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 0.
I20260928 10:58:41.704509 11910 leader_election.cc:302] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 1643b9dc5ccc4ccc8c62bed9fbb07fde; no voters:
I20260928 10:58:41.704787 12260 raft_consensus.cc:2781] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:41.704913 12260 raft_consensus.cc:482] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:41.704996 12260 raft_consensus.cc:3037] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:41.706558 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fb5e6e3e5d436d90543a9ee556ee38" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e" is_pre_election: true
I20260928 10:58:41.706794 11840 raft_consensus.cc:2454] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 0.
I20260928 10:58:41.708308 12260 raft_consensus.cc:504] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.709380 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fb5e6e3e5d436d90543a9ee556ee38" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e"
I20260928 10:58:41.709586 11840 raft_consensus.cc:3037] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:41.709751 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fb5e6e3e5d436d90543a9ee556ee38" candidate_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f50e9a50cc4466a82901ba5881f005"
I20260928 10:58:41.710002 12122 raft_consensus.cc:3037] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:41.712571 11840 raft_consensus.cc:2454] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 1.
I20260928 10:58:41.712893 12122 raft_consensus.cc:2454] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in term 1.
I20260928 10:58:41.713307 11910 leader_election.cc:302] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 1643b9dc5ccc4ccc8c62bed9fbb07fde; no voters:
I20260928 10:58:41.713717 12256 raft_consensus.cc:2781] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:41.713905 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.703580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:37967 (local address 127.3.252.65:37635)
0928 10:58:41.705161 (+ 1581us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:41.705180 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:41.705367 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:41.705924 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:41.705929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:41.705953 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:41.707172 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.707183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.708647 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.708666 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.710807 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.710821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.712681 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.712688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.712865 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.712899 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:58:41.713349 (+ 450us) client_negotiation.cc:240] Negotiation successful
0928 10:58:41.713494 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1443,"mutex_wait_us":361,"thread_start_us":2503,"threads_started":1}
I20260928 10:58:41.713899 12256 raft_consensus.cc:686] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 LEADER]: Becoming Leader. State: Replica: 1643b9dc5ccc4ccc8c62bed9fbb07fde, State: Running, Role: LEADER
I20260928 10:58:41.714464 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.704043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:37635 (local address 127.3.252.66:37967)
0928 10:58:41.704232 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.704242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.705261 (+ 1019us) server_negotiation.cc:407] Connection header received
0928 10:58:41.705530 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.705535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.705610 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.705694 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:41.707345 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.708509 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.710986 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.712284 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.712424 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.712497 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.713051 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.713201 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:58:41.714216 12256 consensus_queue.cc:237] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.715013 12260 leader_election.cc:288] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [CANDIDATE]: Term 1 election: Requested vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355)
I20260928 10:58:41.715974 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fb5e6e3e5d436d90543a9ee556ee38" candidate_uuid: "4718b548e0b04bcbbefee76d5642cf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" is_pre_election: true
I20260928 10:58:41.718508 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.704190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:38967 (local address 127.3.252.67:41355)
0928 10:58:41.704389 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.704395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.705479 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 10:58:41.705694 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.705699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.705762 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.705819 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:58:41.707330 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.708424 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.715230 (+ 6806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.716351 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.716472 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.716539 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.718117 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.718274 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:58:41.719187 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.703694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:41355 (local address 127.3.252.65:38967)
0928 10:58:41.705404 (+ 1710us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:41.705419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:41.705555 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:41.705997 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:41.706001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:41.706035 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:58:41.707016 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.707029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.708592 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.708597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.714856 (+ 6259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:41.714870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.716956 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.716961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:41.717927 (+ 966us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.717966 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:58:41.718762 (+ 796us) client_negotiation.cc:240] Negotiation successful
0928 10:58:41.718912 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1611,"mutex_wait_us":3852,"thread_start_us":1177,"threads_started":1}
I20260928 10:58:41.719131 11681 catalog_manager.cc:5495] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde reported cstate change: term changed from 0 to 1, leader changed from <none> to 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66). New cstate: current_term: 1 leader_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } health_report { overall_health: UNKNOWN } } }
I20260928 10:58:41.719698 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fb5e6e3e5d436d90543a9ee556ee38" candidate_uuid: "4718b548e0b04bcbbefee76d5642cf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f50e9a50cc4466a82901ba5881f005" is_pre_election: true
I20260928 10:58:41.719944 12122 raft_consensus.cc:2379] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4718b548e0b04bcbbefee76d5642cf9e in current term 1: Already voted for candidate 1643b9dc5ccc4ccc8c62bed9fbb07fde in this term.
I20260928 10:58:41.720428 11770 leader_election.cc:302] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4718b548e0b04bcbbefee76d5642cf9e; no voters: 02f50e9a50cc4466a82901ba5881f005, 1643b9dc5ccc4ccc8c62bed9fbb07fde
I20260928 10:58:41.721164 12198 raft_consensus.cc:2726] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 10:58:41.765564 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:41.759968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.3.252.66:37967)
0928 10:58:41.760126 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:41.760130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:41.760152 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:58:41.760383 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:41.760387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:41.760474 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:41.760546 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:41.760978 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.761933 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.763887 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:41.764334 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:41.764435 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:41.764924 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:41.765041 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:41.765102 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:41.765239 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 10:58:41.765302 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:58:41.768450 11840 raft_consensus.cc:1261] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Refusing update from remote peer 1643b9dc5ccc4ccc8c62bed9fbb07fde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:41.768419 12122 raft_consensus.cc:1261] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Refusing update from remote peer 1643b9dc5ccc4ccc8c62bed9fbb07fde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:41.769068 12256 consensus_queue.cc:1035] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [LEADER]: Connected to new peer: Peer: permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:41.769382 12260 consensus_queue.cc:1035] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:41.776953 12243 mvcc.cc:205] Tried to move back new op lower bound from 7334269426757152768 to 7334269426544685056. Current Snapshot: MvccSnapshot[applied={T|T < 7334269426757152768}]
I20260928 10:58:41.785581 11681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:41.785934 11681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:41.789255 11681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Sending DeleteTablet for 3 replicas of tablet f8fb5e6e3e5d436d90543a9ee556ee38
I20260928 10:58:41.790277 11820 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fb5e6e3e5d436d90543a9ee556ee38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:41 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55612
I20260928 10:58:41.790287 11959 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fb5e6e3e5d436d90543a9ee556ee38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:41 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55532
I20260928 10:58:41.790450 12098 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fb5e6e3e5d436d90543a9ee556ee38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:41 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:34490
I20260928 10:58:41.790558 12266 tablet_replica.cc:330] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: stopping tablet replica
I20260928 10:58:41.790572 12097 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fb5e6e3e5d436d90543a9ee556ee38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:41 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:34490
I20260928 10:58:41.790750 12266 raft_consensus.cc:2229] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:41.791059 12266 raft_consensus.cc:2258] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:41.791227 12268 tablet_replica.cc:330] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: stopping tablet replica
I20260928 10:58:41.791415 12268 raft_consensus.cc:2229] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:41.791630 12268 raft_consensus.cc:2258] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:41.791890 12266 ts_tablet_manager.cc:1905] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:41.791910 12267 tablet_replica.cc:330] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: stopping tablet replica
I20260928 10:58:41.792094 12267 raft_consensus.cc:2229] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:41.792300 12267 raft_consensus.cc:2258] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:41.792608 12268 ts_tablet_manager.cc:1905] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:41.793964 12267 ts_tablet_manager.cc:1905] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:41.797888 11681 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 10:58:41.799078 12268 ts_tablet_manager.cc:1918] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:41.799285 12268 log.cc:1196] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/f8fb5e6e3e5d436d90543a9ee556ee38
W20260928 10:58:41.799300 11681 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:58:41.799787 12268 ts_tablet_manager.cc:1939] T f8fb5e6e3e5d436d90543a9ee556ee38 P 02f50e9a50cc4466a82901ba5881f005: Deleting consensus metadata
I20260928 10:58:41.800206 12267 ts_tablet_manager.cc:1918] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:41.800356 12267 log.cc:1196] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/f8fb5e6e3e5d436d90543a9ee556ee38
I20260928 10:58:41.800768 12267 ts_tablet_manager.cc:1939] T f8fb5e6e3e5d436d90543a9ee556ee38 P 4718b548e0b04bcbbefee76d5642cf9e: Deleting consensus metadata
I20260928 10:58:41.801118 11667 catalog_manager.cc:4841] TS 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355): tablet f8fb5e6e3e5d436d90543a9ee556ee38 (table test-table [id=8686ef54960c48dca683e1d906d76209]) successfully deleted
W20260928 10:58:41.801599 11667 catalog_manager.cc:4808] TS 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355): delete failed for tablet f8fb5e6e3e5d436d90543a9ee556ee38 because the tablet was not found. No further retry: Not found: Tablet not found: f8fb5e6e3e5d436d90543a9ee556ee38
I20260928 10:58:41.802273 11667 catalog_manager.cc:4841] TS 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451): tablet f8fb5e6e3e5d436d90543a9ee556ee38 (table test-table [id=8686ef54960c48dca683e1d906d76209]) successfully deleted
I20260928 10:58:41.802871 12266 ts_tablet_manager.cc:1918] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:58:41.803040 12266 log.cc:1196] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/f8fb5e6e3e5d436d90543a9ee556ee38
I20260928 10:58:41.803474 12266 ts_tablet_manager.cc:1939] T f8fb5e6e3e5d436d90543a9ee556ee38 P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting consensus metadata
I20260928 10:58:41.804489 11670 catalog_manager.cc:4841] TS 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967): tablet f8fb5e6e3e5d436d90543a9ee556ee38 (table test-table [id=8686ef54960c48dca683e1d906d76209]) successfully deleted
I20260928 10:58:41.807730 11820 tablet_service.cc:1467] Processing CreateTablet for tablet d0fcd70b52fe44f6911f21d05ace23ed (DEFAULT_TABLE table=test-table [id=9a1d177a8e2f488f858ee7f39deaaf1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:41.807838 11959 tablet_service.cc:1467] Processing CreateTablet for tablet d0fcd70b52fe44f6911f21d05ace23ed (DEFAULT_TABLE table=test-table [id=9a1d177a8e2f488f858ee7f39deaaf1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:41.808058 11820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fcd70b52fe44f6911f21d05ace23ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:41.808254 11959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fcd70b52fe44f6911f21d05ace23ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:41.812077 12285 tablet_bootstrap.cc:492] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: Bootstrap starting.
I20260928 10:58:41.813421 12285 tablet_bootstrap.cc:654] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:41.814677 12097 tablet_service.cc:1467] Processing CreateTablet for tablet d0fcd70b52fe44f6911f21d05ace23ed (DEFAULT_TABLE table=test-table [id=9a1d177a8e2f488f858ee7f39deaaf1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:58:41.815153 12097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fcd70b52fe44f6911f21d05ace23ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:58:41.818341 12282 tablet_bootstrap.cc:492] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Bootstrap starting.
I20260928 10:58:41.819952 12282 tablet_bootstrap.cc:654] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:41.821460 12285 tablet_bootstrap.cc:492] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: No bootstrap required, opened a new log
I20260928 10:58:41.821681 12285 ts_tablet_manager.cc:1397] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260928 10:58:41.822536 12285 raft_consensus.cc:348] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.822808 12285 raft_consensus.cc:374] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:41.822878 12285 raft_consensus.cc:729] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4718b548e0b04bcbbefee76d5642cf9e, State: Initialized, Role: FOLLOWER
I20260928 10:58:41.823101 12285 consensus_queue.cc:260] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.823240 12283 tablet_bootstrap.cc:492] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: Bootstrap starting.
I20260928 10:58:41.823747 12285 ts_tablet_manager.cc:1428] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:41.824874 12283 tablet_bootstrap.cc:654] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:41.825301 12282 tablet_bootstrap.cc:492] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: No bootstrap required, opened a new log
I20260928 10:58:41.825493 12282 ts_tablet_manager.cc:1397] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 10:58:41.826342 12282 raft_consensus.cc:348] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.826607 12282 raft_consensus.cc:374] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:41.826711 12282 raft_consensus.cc:729] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1643b9dc5ccc4ccc8c62bed9fbb07fde, State: Initialized, Role: FOLLOWER
I20260928 10:58:41.826984 12282 consensus_queue.cc:260] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.827608 12282 ts_tablet_manager.cc:1428] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:41.829532 12283 tablet_bootstrap.cc:492] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: No bootstrap required, opened a new log
I20260928 10:58:41.829696 12283 ts_tablet_manager.cc:1397] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260928 10:58:41.830458 12283 raft_consensus.cc:348] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.830710 12283 raft_consensus.cc:374] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:41.830801 12283 raft_consensus.cc:729] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Initialized, Role: FOLLOWER
I20260928 10:58:41.831043 12283 consensus_queue.cc:260] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:41.831594 12283 ts_tablet_manager.cc:1428] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:58:42.029811 12201 raft_consensus.cc:482] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:58:42.030035 12201 raft_consensus.cc:504] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:42.030684 12201 leader_election.cc:288] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967)
I20260928 10:58:42.031220 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fcd70b52fe44f6911f21d05ace23ed" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" is_pre_election: true
I20260928 10:58:42.031265 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fcd70b52fe44f6911f21d05ace23ed" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e" is_pre_election: true
I20260928 10:58:42.031431 11979 raft_consensus.cc:2454] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 0.
I20260928 10:58:42.031528 11840 raft_consensus.cc:2454] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 0.
I20260928 10:58:42.031849 12052 leader_election.cc:302] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 1643b9dc5ccc4ccc8c62bed9fbb07fde; no voters:
I20260928 10:58:42.032115 12201 raft_consensus.cc:2781] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:58:42.032227 12201 raft_consensus.cc:482] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:58:42.032315 12201 raft_consensus.cc:3037] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:42.034412 12201 raft_consensus.cc:504] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:42.034940 12201 leader_election.cc:288] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 election: Requested vote from peers 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451), 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967)
I20260928 10:58:42.035463 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fcd70b52fe44f6911f21d05ace23ed" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4718b548e0b04bcbbefee76d5642cf9e"
I20260928 10:58:42.035465 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fcd70b52fe44f6911f21d05ace23ed" candidate_uuid: "02f50e9a50cc4466a82901ba5881f005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde"
I20260928 10:58:42.035692 11840 raft_consensus.cc:3037] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:42.035738 11979 raft_consensus.cc:3037] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:42.038642 11840 raft_consensus.cc:2454] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 1.
I20260928 10:58:42.038642 11979 raft_consensus.cc:2454] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f50e9a50cc4466a82901ba5881f005 in term 1.
I20260928 10:58:42.039135 12049 leader_election.cc:302] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f50e9a50cc4466a82901ba5881f005, 4718b548e0b04bcbbefee76d5642cf9e; no voters:
I20260928 10:58:42.039474 12201 raft_consensus.cc:2781] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:42.039812 12201 raft_consensus.cc:686] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 1 LEADER]: Becoming Leader. State: Replica: 02f50e9a50cc4466a82901ba5881f005, State: Running, Role: LEADER
I20260928 10:58:42.040066 12201 consensus_queue.cc:237] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } }
I20260928 10:58:42.042531 11681 catalog_manager.cc:5495] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "02f50e9a50cc4466a82901ba5881f005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f50e9a50cc4466a82901ba5881f005" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 41355 } health_report { overall_health: HEALTHY } } }
I20260928 10:58:42.074898 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:42.070514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.3.252.67:41355)
0928 10:58:42.071009 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:42.071013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:42.071027 (+ 14us) server_negotiation.cc:407] Connection header received
0928 10:58:42.071077 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:42.071081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:42.071140 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:42.071205 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:42.071563 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.072258 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.073476 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.073879 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.073964 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:42.074349 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:42.074474 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:42.074537 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:42.074630 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 10:58:42.074691 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
I20260928 10:58:42.087620 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:42.082712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:56027 (local address 127.3.252.67:41355)
0928 10:58:42.082937 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:42.082944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:42.083267 (+ 323us) server_negotiation.cc:407] Connection header received
0928 10:58:42.083429 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:42.083433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:42.083488 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:42.083557 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:58:42.083988 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.084808 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.086115 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.086500 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.086603 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:42.086929 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:58:42.087025 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:58:42.087096 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:42.087215 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 10:58:42.087281 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:58:42.087636 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:42.082534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:41355 (local address 127.3.252.67:56027)
0928 10:58:42.083171 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:42.083186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:42.083319 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:42.083579 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:42.083582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:42.083593 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:58:42.083837 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:42.083846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.084935 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.084940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:42.085994 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:42.086005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.086634 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.086640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:42.086754 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:42.086785 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:58:42.087111 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:58:42.087121 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 10:58:42.087193 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 10:58:42.087322 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":427,"threads_started":1}
I20260928 10:58:42.095517 11979 raft_consensus.cc:1261] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Refusing update from remote peer 02f50e9a50cc4466a82901ba5881f005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:42.095517 11840 raft_consensus.cc:1261] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Refusing update from remote peer 02f50e9a50cc4466a82901ba5881f005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:58:42.096149 12201 consensus_queue.cc:1035] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4718b548e0b04bcbbefee76d5642cf9e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:42.096446 12297 consensus_queue.cc:1035] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1643b9dc5ccc4ccc8c62bed9fbb07fde" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:58:42.138226 11683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:42.138453 11683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:58:42.141666 11683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2cb27af48cf4c91a42eb62be337aa84: Sending DeleteTablet for 3 replicas of tablet d0fcd70b52fe44f6911f21d05ace23ed
I20260928 10:58:42.143217 11820 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fcd70b52fe44f6911f21d05ace23ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:42 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55612
I20260928 10:58:42.143419 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 11754
I20260928 10:58:42.143538 12267 tablet_replica.cc:330] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: stopping tablet replica
I20260928 10:58:42.143726 12267 raft_consensus.cc:2229] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:42.143802 11959 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fcd70b52fe44f6911f21d05ace23ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:42 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:55532
I20260928 10:58:42.143936 12267 raft_consensus.cc:2258] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:42.144078 12266 tablet_replica.cc:330] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: stopping tablet replica
I20260928 10:58:42.144232 12266 raft_consensus.cc:2229] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:58:42.144438 12266 raft_consensus.cc:2258] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:42.144961 12267 ts_tablet_manager.cc:1905] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:42.145506 12266 ts_tablet_manager.cc:1905] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:42.145915 12098 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fcd70b52fe44f6911f21d05ace23ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:58:42 UTC) from {username='oryx', principal='oryx/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:34490
I20260928 10:58:42.146188 12268 tablet_replica.cc:330] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: stopping tablet replica
I20260928 10:58:42.146343 12268 raft_consensus.cc:2229] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:58:42.146639 12268 raft_consensus.cc:2258] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:58:42.147753 12268 ts_tablet_manager.cc:1905] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:58:42.152431 12266 ts_tablet_manager.cc:1918] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:58:42.152575 12266 log.cc:1196] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/d0fcd70b52fe44f6911f21d05ace23ed
I20260928 10:58:42.152985 12266 ts_tablet_manager.cc:1939] T d0fcd70b52fe44f6911f21d05ace23ed P 1643b9dc5ccc4ccc8c62bed9fbb07fde: Deleting consensus metadata
I20260928 10:58:42.155160 11670 catalog_manager.cc:4841] TS 1643b9dc5ccc4ccc8c62bed9fbb07fde (127.3.252.66:37967): tablet d0fcd70b52fe44f6911f21d05ace23ed (table test-table [id=9a1d177a8e2f488f858ee7f39deaaf1d]) successfully deleted
I20260928 10:58:42.157567 12268 ts_tablet_manager.cc:1918] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:58:42.157714 12268 log.cc:1196] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/d0fcd70b52fe44f6911f21d05ace23ed
I20260928 10:58:42.158198 12268 ts_tablet_manager.cc:1939] T d0fcd70b52fe44f6911f21d05ace23ed P 02f50e9a50cc4466a82901ba5881f005: Deleting consensus metadata
I20260928 10:58:42.159379 11667 catalog_manager.cc:4841] TS 02f50e9a50cc4466a82901ba5881f005 (127.3.252.67:41355): tablet d0fcd70b52fe44f6911f21d05ace23ed (table test-table [id=9a1d177a8e2f488f858ee7f39deaaf1d]) successfully deleted
I20260928 10:58:42.160295 12267 ts_tablet_manager.cc:1918] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:58:42.160429 12267 log.cc:1196] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/d0fcd70b52fe44f6911f21d05ace23ed
I20260928 10:58:42.160838 12267 ts_tablet_manager.cc:1939] T d0fcd70b52fe44f6911f21d05ace23ed P 4718b548e0b04bcbbefee76d5642cf9e: Deleting consensus metadata
I20260928 10:58:42.161919 11667 catalog_manager.cc:4841] TS 4718b548e0b04bcbbefee76d5642cf9e (127.3.252.65:40451): tablet d0fcd70b52fe44f6911f21d05ace23ed (table test-table [id=9a1d177a8e2f488f858ee7f39deaaf1d]) successfully deleted
Sep 28 10:58:42 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.65@KRBTEST.COM
I20260928 10:58:42.171696 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:42.145589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.3.252.65:40451)
0928 10:58:42.156972 (+ 11383us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:42.156977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:42.156993 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:58:42.157042 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:42.157045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:42.157103 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:42.157192 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:42.158958 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.160005 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.161810 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.162380 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.162502 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:42.166435 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:42.166456 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:42.166478 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:42.166534 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:42.169281 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:42.169968 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:42.169975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:42.169977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:42.170096 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:42.170399 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:42.170403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:42.170406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:42.170769 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:42.170991 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:42.171259 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 10:58:42.171422 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11263,"thread_start_us":11749,"threads_started":1}
I20260928 10:58:42.172340 11883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:42.270329 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 11754
I20260928 10:58:42.285879 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 11894
Sep 28 10:58:42 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.66@KRBTEST.COM
I20260928 10:58:42.299860 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:42.286572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.3.252.66:37967)
0928 10:58:42.287007 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:42.287012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:42.287028 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:58:42.287079 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:42.287083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:42.287139 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:42.287246 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:42.288372 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.289071 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.290357 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.290759 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.290867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:42.294685 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:42.294700 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:42.294718 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:42.294759 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:42.297388 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:42.298002 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:42.298009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:42.298011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:42.298125 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:42.298483 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:42.298487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:42.298490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:42.298864 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:42.299088 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:42.299315 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 10:58:42.299482 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":250,"threads_started":1}
I20260928 10:58:42.300436 12024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:42.420289 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 11894
I20260928 10:58:42.438362 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12033
Sep 28 10:58:42 dist-test-slave-djjw krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.252.67@KRBTEST.COM
I20260928 10:58:42.452728 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:42.439098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.3.252.67:41355)
0928 10:58:42.439282 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:42.439287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:42.439308 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:58:42.439432 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:42.439437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:42.439508 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:42.439590 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:42.440811 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.441505 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.442789 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:42.443187 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:42.443304 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:42.447568 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:42.447583 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:42.447599 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:42.447635 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:42.450149 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:42.450804 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:42.450812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:42.450814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:42.450945 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:42.451286 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:42.451291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:42.451293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:42.451630 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:42.451943 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:42.452233 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 10:58:42.452441 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:58:42.453382 12163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:42.594058 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 12033
I20260928 10:58:42.614132 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 11651
I20260928 10:58:42.615305 11723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:42.728238 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 11651
2026-09-28T10:58:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7071 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:42 dist-test-slave-djjw krb5kdc[12315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:42 dist-test-slave-djjw krb5kdc[12315](info): set up 2 sockets
Sep 28 10:58:42 dist-test-slave-djjw krb5kdc[12315](info): commencing operation
krb5kdc: starting...
W20260928 10:58:43.735232 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:43 dist-test-slave-djjw krb5kdc[12315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:43.947306 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:35023
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:35023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:44.261624 12328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:44.261967 12328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:44.262102 12328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:44.269304 12328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:44.269418 12328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:44.269490 12328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:44.269562 12328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:44.269626 12328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:44.279376 12328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:35023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:44.280660 12328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:44.281970 12328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:44.293812 12333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:44.293771 12336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:44.293859 12334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:44.294070 12328 server_base.cc:1034] running on GCE node
I20260928 10:58:44.295121 12328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:44.296667 12328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:44.297921 12328 hybrid_clock.cc:648] HybridClock initialized: now 1790593124297863 us; error 49 us; skew 500 ppm
I20260928 10:58:44.298276 12328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T10:58:44Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1638 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:58:44 dist-test-slave-djjw krb5kdc[12342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:58:44 dist-test-slave-djjw krb5kdc[12342](info): set up 2 sockets
Sep 28 10:58:44 dist-test-slave-djjw krb5kdc[12342](info): commencing operation
krb5kdc: starting...
W20260928 10:58:45.371070 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:58:45 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:58:45.579392 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38533
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:45.903822 12355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:45.904115 12355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:45.904237 12355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:45.911408 12355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:45.911518 12355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:45.911573 12355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:45.911650 12355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:45.911705 12355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:45.921327 12355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:45.922655 12355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:45.923959 12355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:45.934863 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:45.934953 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:45.934973 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:45.935784 12355 server_base.cc:1034] running on GCE node
I20260928 10:58:45.936543 12355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:45.937860 12355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:45.939069 12355 hybrid_clock.cc:648] HybridClock initialized: now 1790593125939025 us; error 34 us; skew 500 ppm
I20260928 10:58:45.939397 12355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:45 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593125, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:45.942188 12355 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:45.943500 12355 webserver.cc:460] Webserver started at http://127.3.252.126:43565/ using document root <none> and password file <none>
I20260928 10:58:45.944146 12355 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:45.944257 12355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:45.944537 12355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:45.947374 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "b7e0961e0ab74ad188efcdbaabe52c58"
format_stamp: "Formatted at 2026-09-28 10:58:45 on dist-test-slave-djjw"
server_key: "37b73ccf2e84f8f6cc8ae1c3a8c546b7"
server_key_iv: "c2d8d153192a14f7eb1326650d1a5a98"
server_key_version: "encryptionkey@0"
I20260928 10:58:45.948117 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "b7e0961e0ab74ad188efcdbaabe52c58"
format_stamp: "Formatted at 2026-09-28 10:58:45 on dist-test-slave-djjw"
server_key: "37b73ccf2e84f8f6cc8ae1c3a8c546b7"
server_key_iv: "c2d8d153192a14f7eb1326650d1a5a98"
server_key_version: "encryptionkey@0"
I20260928 10:58:45.953097 12355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:58:45.956600 12370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:45.958004 12355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:45.958165 12355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "b7e0961e0ab74ad188efcdbaabe52c58"
format_stamp: "Formatted at 2026-09-28 10:58:45 on dist-test-slave-djjw"
server_key: "37b73ccf2e84f8f6cc8ae1c3a8c546b7"
server_key_iv: "c2d8d153192a14f7eb1326650d1a5a98"
server_key_version: "encryptionkey@0"
I20260928 10:58:45.958367 12355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:45.984699 12355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:45.988574 12355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:45.988829 12355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:46.005821 12355 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:38533
I20260928 10:58:46.005868 12421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:38533 every 8 connection(s)
I20260928 10:58:46.007154 12355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:46.008586 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 12355
I20260928 10:58:46.008752 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:58:46.009332 4081 external_mini_cluster.cc:1428] Setting key 1d9d16e504aed2dce6a0cbe982ef6c9d
I20260928 10:58:46.011238 12422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:58:46 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:46.023901 12422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Bootstrap starting.
I20260928 10:58:46.027025 12422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Neither blocks nor log segments found. Creating new log.
I20260928 10:58:46.028190 12422 log.cc:824] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:46.031003 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:46.011174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.3.252.126:38533)
0928 10:58:46.011915 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:46.011926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:46.011965 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:58:46.013473 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:46.013494 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:46.013956 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:46.014295 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:46.016013 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:46.017366 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:46.019067 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:46.019701 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:46.019834 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:46.024248 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:46.024256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:46.024271 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:46.024305 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:46.027066 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:46.027819 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:46.027825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:46.027831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:46.028025 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:46.028415 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:46.028419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:46.028421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:46.028963 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:46.029243 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:46.029662 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 10:58:46.030069 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":332,"threads_started":1}
I20260928 10:58:46.031059 12422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: No bootstrap required, opened a new log
I20260928 10:58:46.038429 12422 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:46.038803 12422 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:58:46.038906 12422 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e0961e0ab74ad188efcdbaabe52c58, State: Initialized, Role: FOLLOWER
I20260928 10:58:46.039425 12422 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:46.039635 12422 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:46.039750 12422 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:46.039896 12422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:58:46.041975 12422 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:46.042456 12422 leader_election.cc:302] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e0961e0ab74ad188efcdbaabe52c58; no voters:
I20260928 10:58:46.043033 12422 leader_election.cc:288] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:58:46.043186 12427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:58:46.044180 12427 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 LEADER]: Becoming Leader. State: Replica: b7e0961e0ab74ad188efcdbaabe52c58, State: Running, Role: LEADER
I20260928 10:58:46.044777 12427 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:46.045151 12422 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:46.047614 12428 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } } }
I20260928 10:58:46.047590 12429 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e0961e0ab74ad188efcdbaabe52c58. Latest consensus state: current_term: 1 leader_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } } }
I20260928 10:58:46.048007 12429 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:46.048004 12428 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:46.050365 12435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:46.055565 12435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:46.064528 12435 catalog_manager.cc:1347] Generated new cluster ID: 99a49e25f87749f4835f28eeb55c63f6
I20260928 10:58:46.064652 12435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:46.074266 12435 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:58:46.075591 12435 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:46.085369 12435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Generated new TSK 0
I20260928 10:58:46.086304 12435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:58:46.149660 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38533
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:46.472422 12448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:46.472695 12448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:46.472823 12448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:46.480268 12448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:46.480420 12448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:46.480607 12448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:46.490371 12448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:46.491679 12448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:46.492969 12448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:46.505256 12453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:46.505283 12456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:46.505319 12454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:46.506376 12448 server_base.cc:1034] running on GCE node
I20260928 10:58:46.507018 12448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:46.507988 12448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:46.509193 12448 hybrid_clock.cc:648] HybridClock initialized: now 1790593126509146 us; error 43 us; skew 500 ppm
I20260928 10:58:46.509517 12448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:46 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593126, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:46.512492 12448 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:58:46.514003 12448 webserver.cc:460] Webserver started at http://127.3.252.65:33347/ using document root <none> and password file <none>
I20260928 10:58:46.514635 12448 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:46.514726 12448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:46.514977 12448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:46.517778 12448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "5dbfec934b474c1a894eca9e03915a99"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "9a0c415584b7b6e04fff9e2d706017ce"
server_key_iv: "dd38bea09995d2f95ad8dd6baa6cdce5"
server_key_version: "encryptionkey@0"
I20260928 10:58:46.518553 12448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "5dbfec934b474c1a894eca9e03915a99"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "9a0c415584b7b6e04fff9e2d706017ce"
server_key_iv: "dd38bea09995d2f95ad8dd6baa6cdce5"
server_key_version: "encryptionkey@0"
I20260928 10:58:46.523403 12448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:58:46.526932 12463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:46.528275 12448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:58:46.528424 12448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "5dbfec934b474c1a894eca9e03915a99"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "9a0c415584b7b6e04fff9e2d706017ce"
server_key_iv: "dd38bea09995d2f95ad8dd6baa6cdce5"
server_key_version: "encryptionkey@0"
I20260928 10:58:46.528617 12448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:46.539564 12448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:46.543361 12448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:46.543615 12448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:46.544519 12448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:46.545792 12448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:46.545897 12448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:46.545998 12448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:46.546041 12448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:46.575519 12448 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:41081
I20260928 10:58:46.575552 12575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:41081 every 8 connection(s)
I20260928 10:58:46.576728 12448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:46.583228 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 12448
I20260928 10:58:46.583408 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:58:46.584005 4081 external_mini_cluster.cc:1428] Setting key b0266b7fae9d9cca65d5b4075a4a3de4
Sep 28 10:58:46 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593126, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:46.596117 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:46.579057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:47469 (local address 127.3.252.126:38533)
0928 10:58:46.579447 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:46.579451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:46.580276 (+ 825us) server_negotiation.cc:407] Connection header received
0928 10:58:46.582079 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:46.582085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:46.582158 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:46.582251 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:46.583767 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:46.584520 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:46.586468 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:46.587027 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:46.587146 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:46.590539 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:46.590545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:46.590549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:46.590585 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:46.593151 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:46.594042 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:46.594047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:46.594049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:46.594167 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:46.594583 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:46.594586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:46.594588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:46.594842 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:46.594952 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:46.595705 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 10:58:46.595866 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":213,"threads_started":1}
I20260928 10:58:46.597458 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:46.579505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:47469)
0928 10:58:46.580129 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:46.580170 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:46.581774 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:46.582486 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:46.582494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:46.583006 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:46.583602 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:46.583621 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:46.584664 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:46.584668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:46.586328 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:46.586346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:46.587217 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:46.587222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:46.587461 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:46.588096 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:46.588118 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:46.590321 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:46.593291 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:46.593301 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:46.593319 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:46.593913 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:46.594258 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:46.594263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:46.594267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:46.594494 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:46.594955 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:46.594963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:46.595359 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 10:58:46.595650 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 10:58:46.596159 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":246,"threads_started":1}
I20260928 10:58:46.598981 12576 heartbeater.cc:347] Connected to a master server at 127.3.252.126:38533
I20260928 10:58:46.599267 12576 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:46.599927 12576 heartbeater.cc:511] Master 127.3.252.126:38533 requested a full tablet report, sending...
I20260928 10:58:46.601876 12387 ts_manager.cc:194] Registered new tserver with Master: 5dbfec934b474c1a894eca9e03915a99 (127.3.252.65:41081)
I20260928 10:58:46.603256 12387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:47469
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:58:46.632649 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38533
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:46.945003 12582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:46.945326 12582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:46.945484 12582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:46.952700 12582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:46.952824 12582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:46.952986 12582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:46.962743 12582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:46.964063 12582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:46.965358 12582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:46.978621 12588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:46.978734 12590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:46.978621 12587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:46.979254 12582 server_base.cc:1034] running on GCE node
I20260928 10:58:46.979848 12582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:46.980774 12582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:46.981909 12582 hybrid_clock.cc:648] HybridClock initialized: now 1790593126981853 us; error 50 us; skew 500 ppm
I20260928 10:58:46.982198 12582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:46 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593126, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:46.985030 12582 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:58:46.986383 12582 webserver.cc:460] Webserver started at http://127.3.252.66:36687/ using document root <none> and password file <none>
I20260928 10:58:46.987021 12582 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:46.987130 12582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:46.987411 12582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:46.990268 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "3314c794bd56454488c32d2a0e9dfe04"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "d667e2669f3c2badd941d4816f622416"
server_key_iv: "fbd230f56a381117c9811f4fee7b5761"
server_key_version: "encryptionkey@0"
I20260928 10:58:46.990993 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "3314c794bd56454488c32d2a0e9dfe04"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "d667e2669f3c2badd941d4816f622416"
server_key_iv: "fbd230f56a381117c9811f4fee7b5761"
server_key_version: "encryptionkey@0"
I20260928 10:58:46.995893 12582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:46.999243 12597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:47.000528 12582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:58:47.000674 12582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "3314c794bd56454488c32d2a0e9dfe04"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "d667e2669f3c2badd941d4816f622416"
server_key_iv: "fbd230f56a381117c9811f4fee7b5761"
server_key_version: "encryptionkey@0"
I20260928 10:58:47.000851 12582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:47.039431 12582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:47.043685 12582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:47.043974 12582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:47.045020 12582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:47.046551 12582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:47.046627 12582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:47.046746 12582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:47.046797 12582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:47.077085 12582 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:46453
I20260928 10:58:47.077142 12709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:46453 every 8 connection(s)
I20260928 10:58:47.078336 12582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:47.082093 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 12582
I20260928 10:58:47.082268 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:58:47.082832 4081 external_mini_cluster.cc:1428] Setting key fc4dc84cb5160187f36bfeab45480e3c
Sep 28 10:58:47 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593126, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:47.099961 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:47.080697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:53247 (local address 127.3.252.126:38533)
0928 10:58:47.080856 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:47.080887 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:47.082120 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 10:58:47.084330 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:47.084336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:47.084406 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:47.084514 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:47.086481 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.087185 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.089205 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.089621 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.089723 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:47.093693 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:47.093699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:47.093704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:47.093742 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:47.096724 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.097572 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.097578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.097581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.097743 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.098258 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.098263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.098265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.098585 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:47.098717 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:47.099411 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 10:58:47.099634 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:58:47.100864 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:47.081180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:53247)
0928 10:58:47.081932 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:47.081991 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:47.084069 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:47.084741 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:47.084749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:47.085479 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:47.086277 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:47.086302 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.087324 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.087329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:47.089057 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:47.089071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.089884 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.089891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:47.090172 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:47.090868 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:47.090901 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:47.093451 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:47.096917 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:47.096925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:47.096940 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:47.097440 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:47.097916 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:47.097920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:47.097922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:47.098138 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:47.098706 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:47.098713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:47.099067 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 10:58:47.099300 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 10:58:47.099718 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":250,"threads_started":1}
I20260928 10:58:47.102257 12710 heartbeater.cc:347] Connected to a master server at 127.3.252.126:38533
I20260928 10:58:47.102569 12710 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:47.103406 12710 heartbeater.cc:511] Master 127.3.252.126:38533 requested a full tablet report, sending...
I20260928 10:58:47.104918 12387 ts_manager.cc:194] Registered new tserver with Master: 3314c794bd56454488c32d2a0e9dfe04 (127.3.252.66:46453)
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 10:58:47.105820 12387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:53247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:58:47.133489 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38533
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:47.452651 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:47.452947 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:47.453078 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:47.460482 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:47.460614 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:47.460762 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:47.470790 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:47.472102 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:47.473389 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:47.485966 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:47.485975 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:47.485989 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:47.486779 12715 server_base.cc:1034] running on GCE node
I20260928 10:58:47.487332 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:47.488193 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:47.489409 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790593127489352 us; error 41 us; skew 500 ppm
I20260928 10:58:47.489725 12715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:47 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593127, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:47.492640 12715 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:58:47.494071 12715 webserver.cc:460] Webserver started at http://127.3.252.67:40755/ using document root <none> and password file <none>
I20260928 10:58:47.494719 12715 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:47.494830 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:47.495113 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:58:47.497987 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "ee6ff326b7c04c40bd6447b1fbf37044"
format_stamp: "Formatted at 2026-09-28 10:58:47 on dist-test-slave-djjw"
server_key: "ff42d97b4ac0f9a40225c350066b5c95"
server_key_iv: "93748094adb65aa7f220139f93ec85d8"
server_key_version: "encryptionkey@0"
I20260928 10:58:47.498749 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "ee6ff326b7c04c40bd6447b1fbf37044"
format_stamp: "Formatted at 2026-09-28 10:58:47 on dist-test-slave-djjw"
server_key: "ff42d97b4ac0f9a40225c350066b5c95"
server_key_iv: "93748094adb65aa7f220139f93ec85d8"
server_key_version: "encryptionkey@0"
I20260928 10:58:47.503961 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:58:47.507359 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:47.508635 12715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:58:47.508780 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "ee6ff326b7c04c40bd6447b1fbf37044"
format_stamp: "Formatted at 2026-09-28 10:58:47 on dist-test-slave-djjw"
server_key: "ff42d97b4ac0f9a40225c350066b5c95"
server_key_iv: "93748094adb65aa7f220139f93ec85d8"
server_key_version: "encryptionkey@0"
I20260928 10:58:47.508960 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:47.526582 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:47.530308 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:47.530562 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:47.531466 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:47.532829 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:47.532904 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:47.533020 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:47.533071 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:47.563066 12715 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:42703
I20260928 10:58:47.563123 12842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:42703 every 8 connection(s)
I20260928 10:58:47.564339 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:47.574108 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 12715
I20260928 10:58:47.574282 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:58:47.574728 4081 external_mini_cluster.cc:1428] Setting key d568f35160ead38e280fe97a2c4176bf
Sep 28 10:58:47 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593127, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:58:47.583945 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:47.567049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:37547 (local address 127.3.252.126:38533)
0928 10:58:47.567246 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:47.567252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:47.568420 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 10:58:47.570250 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:47.570254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:47.570337 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:47.570428 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:47.571852 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.572572 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.574345 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.574740 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.574877 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:47.578753 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:47.578759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:47.578763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:47.578798 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:47.581106 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.581770 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.581776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.581780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.581945 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.582342 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.582347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.582351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.582612 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:47.582737 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:47.583400 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 10:58:47.583606 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:58:47.584822 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:47.567554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:37547)
0928 10:58:47.568256 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:47.568300 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:47.569958 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:47.570671 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:47.570679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:47.571174 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:47.571699 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:47.571716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.572709 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.572713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:47.574205 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:47.574221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.575151 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.575159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:47.575431 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:47.576132 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:47.576153 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:47.578562 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:58:47.581234 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:47.581244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:47.581259 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:47.581650 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:47.582042 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:58:47.582047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:58:47.582051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:58:47.582226 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:58:47.582729 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:58:47.582736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:58:47.583061 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 10:58:47.583283 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 10:58:47.583780 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":271,"threads_started":1}
I20260928 10:58:47.586081 12843 heartbeater.cc:347] Connected to a master server at 127.3.252.126:38533
I20260928 10:58:47.586341 12843 heartbeater.cc:464] Registering TS with master...
I20260928 10:58:47.587000 12843 heartbeater.cc:511] Master 127.3.252.126:38533 requested a full tablet report, sending...
I20260928 10:58:47.588378 12387 ts_manager.cc:194] Registered new tserver with Master: ee6ff326b7c04c40bd6447b1fbf37044 (127.3.252.67:42703)
I20260928 10:58:47.589207 12387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:37547
I20260928 10:58:47.589921 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:58:47.606400 12576 heartbeater.cc:503] Master 127.3.252.126:38533 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab.
Entry for principal oryx/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab.
WARNING: no policy specified for oryx/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab.
Entry for principal oryx/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab.
WARNING: no policy specified for oryx/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab.
Entry for principal oryx/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab.
I20260928 10:58:47.710774 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12448
Sep 28 10:58:47 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:58:47.725804 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:47.711875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.3.252.65:41081)
0928 10:58:47.712507 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:47.712514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:47.712562 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:58:47.712635 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:47.712644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:47.712860 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:47.712993 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:47.714349 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.715362 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.716574 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.717134 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.717227 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:47.720951 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:47.720962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:47.720977 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:47.721013 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:47.723634 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.724220 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.724227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.724229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.724345 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.724607 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.724610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.724612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.724940 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:47.725149 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:47.725340 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 10:58:47.725533 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":299,"threads_started":1}
I20260928 10:58:47.727116 12571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:47.809208 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 12448
I20260928 10:58:47.822342 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12582
Sep 28 10:58:47 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:58:47.837550 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:47.823182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.3.252.66:46453)
0928 10:58:47.823667 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:47.823674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:47.823699 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:58:47.823778 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:47.823784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:47.823994 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:47.824138 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:47.825316 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.826358 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.827614 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.828152 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.828261 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:47.832259 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:47.832270 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:47.832284 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:47.832316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:47.834955 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.835588 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.835595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.835597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.835719 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.836045 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.836049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.836051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.836378 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:47.836651 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:47.836964 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 10:58:47.837212 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":229,"threads_started":1}
I20260928 10:58:47.838879 12705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:47.918923 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 12582
I20260928 10:58:47.931569 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12715
Sep 28 10:58:47 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:58:47.946427 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:47.932365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.3.252.67:42703)
0928 10:58:47.932954 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:47.932960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:47.932989 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:58:47.933045 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:47.933051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:47.933326 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:47.933484 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:47.934671 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.935660 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.936803 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:47.937332 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:47.937429 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:47.941419 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:47.941437 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:47.941457 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:47.941502 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:47.944106 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.944672 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.944678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.944680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.944781 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:47.945034 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:47.945036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:47.945038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:47.945329 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:47.945596 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:47.945818 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 10:58:47.946141 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":279,"threads_started":1}
I20260928 10:58:47.947654 12838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:48.027937 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 12715
I20260928 10:58:48.040756 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12355
I20260928 10:58:48.041941 12417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:58:48.143167 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 12355
I20260928 10:58:48.160188 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38533
--webserver_interface=127.3.252.126
--webserver_port=43565
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:58:48.474651 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:48.474965 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:48.475092 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:48.482293 12860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:58:48.482405 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:48.482482 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:48.482556 12860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:58:48.482614 12860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:58:48.492434 12860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:48.493700 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:48.494952 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:48.508908 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:48.508926 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:48.508967 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:48.509654 12860 server_base.cc:1034] running on GCE node
I20260928 10:58:48.510460 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:48.511853 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:48.513054 12860 hybrid_clock.cc:648] HybridClock initialized: now 1790593128513003 us; error 39 us; skew 500 ppm
I20260928 10:58:48.513372 12860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:48 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:48.516245 12860 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:58:48.517602 12860 webserver.cc:460] Webserver started at http://127.3.252.126:43565/ using document root <none> and password file <none>
I20260928 10:58:48.518288 12860 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:48.518410 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:48.523603 12860 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 10:58:48.526433 12875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:48.527683 12860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:48.527834 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "b7e0961e0ab74ad188efcdbaabe52c58"
format_stamp: "Formatted at 2026-09-28 10:58:45 on dist-test-slave-djjw"
server_key: "37b73ccf2e84f8f6cc8ae1c3a8c546b7"
server_key_iv: "c2d8d153192a14f7eb1326650d1a5a98"
server_key_version: "encryptionkey@0"
I20260928 10:58:48.528785 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:48.540793 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:48.544814 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:48.545063 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:48.562040 12860 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:38533
I20260928 10:58:48.562062 12926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:38533 every 8 connection(s)
I20260928 10:58:48.563467 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:58:48.569877 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 12860
I20260928 10:58:48.570511 12927 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:58:48.570631 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:41081
--local_ip_for_outbound_sockets=127.3.252.65
--tserver_master_addrs=127.3.252.126:38533
--webserver_port=33347
--webserver_interface=127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:58:48.573765 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Bootstrap starting.
I20260928 10:58:48.591938 12927 log.cc:824] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Log is configured to *not* fsync() on all Append() calls
I20260928 10:58:48.600637 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:58:48.601122 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Bootstrap complete.
I20260928 10:58:48.609863 12927 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:48.610675 12927 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e0961e0ab74ad188efcdbaabe52c58, State: Initialized, Role: FOLLOWER
I20260928 10:58:48.611244 12927 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:48.611461 12927 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:58:48.611562 12927 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:58:48.611709 12927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:58:48.615271 12927 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:48.615732 12927 leader_election.cc:302] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e0961e0ab74ad188efcdbaabe52c58; no voters:
I20260928 10:58:48.616303 12927 leader_election.cc:288] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:58:48.616433 12931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:58:48.617498 12931 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [term 2 LEADER]: Becoming Leader. State: Replica: b7e0961e0ab74ad188efcdbaabe52c58, State: Running, Role: LEADER
I20260928 10:58:48.618031 12931 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } }
I20260928 10:58:48.618465 12927 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:58:48.620268 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e0961e0ab74ad188efcdbaabe52c58. Latest consensus state: current_term: 2 leader_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } } }
I20260928 10:58:48.620630 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:48.621214 12932 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e0961e0ab74ad188efcdbaabe52c58" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38533 } } }
I20260928 10:58:48.621488 12932 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58 [sys.catalog]: This master's current role is: LEADER
I20260928 10:58:48.631843 12937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:58:48.638201 12937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:58:48.639539 12937 catalog_manager.cc:1259] Loaded cluster ID: 99a49e25f87749f4835f28eeb55c63f6
I20260928 10:58:48.639638 12937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:58:48.643667 12937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:58:48.645030 12937 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7e0961e0ab74ad188efcdbaabe52c58: Loaded TSK: 0
I20260928 10:58:48.646736 12937 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:58:48.896580 12929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:48.896855 12929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:48.896986 12929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:48.904201 12929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:48.904335 12929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:48.904476 12929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:58:48.914204 12929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.65.keytab
--principal=oryx/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:41081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=33347
--tserver_master_addrs=127.3.252.126:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:48.915455 12929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:48.916705 12929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:48.937245 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:48.937217 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:48.937294 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:48.938961 12929 server_base.cc:1034] running on GCE node
I20260928 10:58:48.939604 12929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:48.940565 12929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:48.941766 12929 hybrid_clock.cc:648] HybridClock initialized: now 1790593128941706 us; error 50 us; skew 500 ppm
I20260928 10:58:48.942082 12929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:48 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593128, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:48.945117 12929 init.cc:377] Logged in from keytab as oryx/127.3.252.65@KRBTEST.COM (short username oryx)
I20260928 10:58:48.946552 12929 webserver.cc:460] Webserver started at http://127.3.252.65:33347/ using document root <none> and password file <none>
I20260928 10:58:48.947186 12929 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:48.947305 12929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:48.952484 12929 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 10:58:48.955117 12963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:48.956499 12929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:48.956634 12929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "5dbfec934b474c1a894eca9e03915a99"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "9a0c415584b7b6e04fff9e2d706017ce"
server_key_iv: "dd38bea09995d2f95ad8dd6baa6cdce5"
server_key_version: "encryptionkey@0"
I20260928 10:58:48.957540 12929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:48.981626 12929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:48.985498 12929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:48.985742 12929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:48.986853 12929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:48.988183 12929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:48.988276 12929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:48.988376 12929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:48.988453 12929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:49.017554 12929 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:41081
I20260928 10:58:49.017618 13075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:41081 every 8 connection(s)
I20260928 10:58:49.019023 12929 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:58:49.020776 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 12929
I20260928 10:58:49.021394 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:46453
--local_ip_for_outbound_sockets=127.3.252.66
--tserver_master_addrs=127.3.252.126:38533
--webserver_port=36687
--webserver_interface=127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:58:49 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:49 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:58:49.357044 13080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:49.357366 13080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:49.357491 13080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:49.365087 13080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:49.365226 13080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:49.365389 13080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:58:49.375314 13080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.66.keytab
--principal=oryx/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:46453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=36687
--tserver_master_addrs=127.3.252.126:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:49.376631 13080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:49.377904 13080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:49.390767 13087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:49.391218 13086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:49.391530 13089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:49.391957 13080 server_base.cc:1034] running on GCE node
I20260928 10:58:49.392516 13080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:49.393477 13080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:49.394667 13080 hybrid_clock.cc:648] HybridClock initialized: now 1790593129394589 us; error 76 us; skew 500 ppm
I20260928 10:58:49.394984 13080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:49 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593129, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:49.397977 13080 init.cc:377] Logged in from keytab as oryx/127.3.252.66@KRBTEST.COM (short username oryx)
I20260928 10:58:49.399396 13080 webserver.cc:460] Webserver started at http://127.3.252.66:36687/ using document root <none> and password file <none>
I20260928 10:58:49.400028 13080 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:49.400141 13080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:49.405715 13080 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:58:49.408591 13096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:49.410112 13080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:58:49.410254 13080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "3314c794bd56454488c32d2a0e9dfe04"
format_stamp: "Formatted at 2026-09-28 10:58:46 on dist-test-slave-djjw"
server_key: "d667e2669f3c2badd941d4816f622416"
server_key_iv: "fbd230f56a381117c9811f4fee7b5761"
server_key_version: "encryptionkey@0"
I20260928 10:58:49.411180 13080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:49.444810 13080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:49.448895 13080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:49.449126 13080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:49.450125 13080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:49.451450 13080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:49.451547 13080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:49.451649 13080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:49.451714 13080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:49.486877 13080 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:46453
I20260928 10:58:49.486899 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:46453 every 8 connection(s)
I20260928 10:58:49.488252 13080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:58:49.494721 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13080
I20260928 10:58:49.495556 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:42703
--local_ip_for_outbound_sockets=127.3.252.67
--tserver_master_addrs=127.3.252.126:38533
--webserver_port=40755
--webserver_interface=127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:58:49 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:49 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:58:49.822353 13213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:58:49.822629 13213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:58:49.822757 13213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:58:49.829910 13213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:58:49.830027 13213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:58:49.830174 13213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:58:49.839975 13213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/krb5kdc/oryx_127.3.252.67.keytab
--principal=oryx/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:42703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=40755
--tserver_master_addrs=127.3.252.126:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:58:49.841248 13213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:58:49.842545 13213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:58:49.856998 13219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:49.858201 13220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:58:49.864193 13222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:58:49.864605 13213 server_base.cc:1034] running on GCE node
I20260928 10:58:49.865212 13213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:58:49.866189 13213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:58:49.867403 13213 hybrid_clock.cc:648] HybridClock initialized: now 1790593129867347 us; error 46 us; skew 500 ppm
I20260928 10:58:49.867689 13213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:58:49 dist-test-slave-djjw krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593129, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:58:49.870743 13213 init.cc:377] Logged in from keytab as oryx/127.3.252.67@KRBTEST.COM (short username oryx)
I20260928 10:58:49.875370 13213 webserver.cc:460] Webserver started at http://127.3.252.67:40755/ using document root <none> and password file <none>
I20260928 10:58:49.876047 13213 fs_manager.cc:359] Metadata directory not provided
I20260928 10:58:49.876174 13213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:58:49.881336 13213 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:58:49.883910 13229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:49.885229 13213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:58:49.885372 13213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "ee6ff326b7c04c40bd6447b1fbf37044"
format_stamp: "Formatted at 2026-09-28 10:58:47 on dist-test-slave-djjw"
server_key: "ff42d97b4ac0f9a40225c350066b5c95"
server_key_iv: "93748094adb65aa7f220139f93ec85d8"
server_key_version: "encryptionkey@0"
I20260928 10:58:49.886312 13213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:58:49.906479 13213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:58:49.910774 13213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:58:49.911026 13213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:58:49.911989 13213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:58:49.913319 13213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:58:49.913410 13213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:49.913516 13213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:58:49.913585 13213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:58:49.952050 13213 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:42703
I20260928 10:58:49.952095 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:42703 every 8 connection(s)
I20260928 10:58:49.953532 13213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:58:49.955737 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13213
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:50.035203 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:49.957129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.3.252.126:38533)
0928 10:58:50.006241 (+ 49112us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:50.006249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:50.009688 (+ 3439us) server_negotiation.cc:407] Connection header received
0928 10:58:50.009769 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:50.009775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:50.009882 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:50.009984 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:50.012859 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.014271 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.020309 (+ 6038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.020992 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.021264 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.026982 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:58:50.026991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:58:50.027007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:58:50.027046 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:58:50.029769 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:50.030809 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:50.030818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:50.030830 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:50.031077 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:58:50.031535 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:58:50.031540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:58:50.031543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:58:50.032349 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:58:50.032701 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 10:58:50.033610 (+ 909us) server_negotiation.cc:299] Negotiation successful
0928 10:58:50.034098 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48964,"thread_start_us":53432,"threads_started":1}
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:58:50.267299 13078 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.234s user 0.000s sys 0.011s
W20260928 10:58:50.268960 13079 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:49.022153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:34579 (local address 127.3.252.126:38533)
0928 10:58:49.026224 (+ 4071us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:49.026237 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:49.026282 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:58:49.026543 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:49.026556 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:49.026883 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:49.027206 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:49.028767 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.030067 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.031609 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.032428 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.032626 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.268656 (+1236030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:34579: BlockingRecv error: recv got EOF from 127.3.252.65:34579 (error 108)
Metrics: {"server-negotiator.queue_time_us":3849,"thread_start_us":3684,"threads_started":1}
I20260928 10:58:50.269734 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:49.022168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:34579)
0928 10:58:49.022937 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:49.022989 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:49.024674 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:49.027409 (+ 2735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:49.027420 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:49.028015 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:49.028574 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:49.028594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.030228 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.030232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:49.031458 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:49.031471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.032653 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.032661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:49.032955 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:49.033634 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:49.033658 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:50.268394 (+1234736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":318,"threads_started":1}
W20260928 10:58:50.270661 13076 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:38533 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.252.126:38533: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:50.288782 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:50.271386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:45629)
0928 10:58:50.272128 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:50.272151 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:50.272340 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:50.272918 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:50.272924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:50.273440 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:50.273695 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.273708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.275419 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.275425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.277141 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.277157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.278519 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.278527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.278735 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.282092 (+ 3357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:50.282121 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:50.288428 (+ 6307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601}
W20260928 10:58:50.289806 13079 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:50.271750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:45629 (local address 127.3.252.126:38533)
0928 10:58:50.271954 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:50.271962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:50.272225 (+ 263us) server_negotiation.cc:407] Connection header received
0928 10:58:50.272493 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:50.272499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:50.272584 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:50.272662 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:50.273939 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.275189 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.277337 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.278076 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.278183 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.289538 (+ 11355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:45629: BlockingRecv error: recv got EOF from 127.3.252.65:45629 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:58:50.306432 13079 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:50.291169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:60357 (local address 127.3.252.126:38533)
0928 10:58:50.291347 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:50.291354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:50.291608 (+ 254us) server_negotiation.cc:407] Connection header received
0928 10:58:50.291887 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:50.291893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:50.291980 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:50.292057 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:50.293189 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.294477 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.298833 (+ 4356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.299575 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.299677 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.306099 (+ 6422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:60357: BlockingRecv error: recv got EOF from 127.3.252.65:60357 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 10:58:50.307054 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:50.290804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:60357)
0928 10:58:50.291520 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:50.291542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:50.291735 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:50.292315 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:50.292320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:50.292775 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:50.293003 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.293016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.294697 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.294703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.298611 (+ 3908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.298630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.299947 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.299954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.300177 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.300934 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:50.300958 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:50.306753 (+ 5795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594}
W20260928 10:58:50.307432 13076 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:58:50.685410 13212 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.186s user 0.003s sys 0.010s
W20260928 10:58:50.686833 13211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:49.490866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:45007 (local address 127.3.252.126:38533)
0928 10:58:49.491245 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:49.491249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:49.492614 (+ 1365us) server_negotiation.cc:407] Connection header received
0928 10:58:49.494399 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:49.494405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:49.494483 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:49.494574 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:49.496119 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.496821 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.498336 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.498775 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.498892 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.686611 (+1187719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:45007: BlockingRecv error: recv got EOF from 127.3.252.66:45007 (error 108)
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":186,"threads_started":1}
I20260928 10:58:50.687764 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:49.491705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:45007)
0928 10:58:49.492466 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:49.492508 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:49.494148 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:49.494811 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:49.494822 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:49.495335 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:49.495959 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:49.495978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.496981 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.496985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:49.498208 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:49.498219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:49.498978 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:49.498987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:49.499205 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:49.499793 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:49.499814 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:50.686468 (+1186654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":304,"threads_started":1}
W20260928 10:58:50.688715 13209 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:38533 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.252.126:38533: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:58:50.699843 13211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:50.689409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:34329 (local address 127.3.252.126:38533)
0928 10:58:50.689555 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:50.689559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:50.689614 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:58:50.689792 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:50.689796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:50.689884 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:50.689998 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:50.691010 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.691746 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.693691 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.694288 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.694382 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.699673 (+ 5291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:34329: BlockingRecv error: recv got EOF from 127.3.252.66:34329 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 10:58:50.699855 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:50.689305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:34329)
0928 10:58:50.689513 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:50.689537 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:50.689686 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:50.690031 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:50.690036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:50.690624 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:50.690854 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.690867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.691872 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.691877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.693557 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.693572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.694477 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.694485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.694684 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.695345 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:50.695369 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:50.699514 (+ 4145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:50 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:50.710259 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:50.700625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:35327)
0928 10:58:50.700810 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:50.700829 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:50.700912 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:50.701254 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:50.701258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:50.701559 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:50.701705 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.701713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.703219 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.703223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.704184 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.704195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.705132 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.705139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.705667 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.706318 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:50.706343 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:50.710058 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 10:58:50.710633 13209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:58:50.711084 13211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:50.700766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:35327 (local address 127.3.252.126:38533)
0928 10:58:50.700937 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:50.700943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:50.700964 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:58:50.701033 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:50.701038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:50.701130 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:50.701239 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:50.701857 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.703081 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.704308 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.704976 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.705098 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.710787 (+ 5689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:35327: BlockingRecv error: recv got EOF from 127.3.252.66:35327 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 10:58:51.038815 13346 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.006s user 0.008s sys 0.004s
W20260928 10:58:51.039939 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:49.977465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:59885 (local address 127.3.252.126:38533)
0928 10:58:50.010545 (+ 33080us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:50.010551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:50.010568 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:58:50.017692 (+ 7124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:50.017699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:50.017794 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:50.017898 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:50.024467 (+ 6569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.025607 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.030764 (+ 5157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.031383 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.031499 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.039700 (+1008201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:59885: BlockingRecv error: recv got EOF from 127.3.252.67:59885 (error 108)
Metrics: {"server-negotiator.queue_time_us":32945,"thread_start_us":28796,"threads_started":1}
I20260928 10:58:51.040532 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:49.976776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:59885)
0928 10:58:50.003843 (+ 27067us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:50.003920 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:50.015964 (+ 12044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:50.018737 (+ 2773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:50.018753 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:50.021737 (+ 2984us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:50.022702 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.022731 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.025817 (+ 3086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.025823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.030629 (+ 4806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:50.030644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:50.031825 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:50.031846 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:50.032136 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:50.032801 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:50.032852 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:51.039530 (+1006678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26621,"thread_start_us":29447,"threads_started":1}
W20260928 10:58:51.041306 13342 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:38533 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.252.126:38533: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:51.053288 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:51.041868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:38287)
0928 10:58:51.042045 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:51.042064 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:51.042171 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:51.042452 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:51.042457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:51.042800 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:51.042957 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.042968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.043917 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.043923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.044860 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.044871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.045525 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.045531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.045684 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.049478 (+ 3794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:51.049496 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:51.052987 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 10:58:51.053387 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:51.041940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:38287 (local address 127.3.252.126:38533)
0928 10:58:51.042080 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:51.042085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:51.042104 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:58:51.042275 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:51.042280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:51.042338 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:51.042413 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:51.043074 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.043786 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.044983 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.045371 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.045473 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.053166 (+ 7693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:38287: BlockingRecv error: recv got EOF from 127.3.252.67:38287 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:51.061240 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:51.053992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:35091)
0928 10:58:51.054122 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:51.054137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:51.054236 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:51.054563 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:51.054568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:51.054859 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:51.055009 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.055018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.055943 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.055948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.056866 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.056876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.057561 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.057566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.057727 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.058217 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:51.058233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:51.061058 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 10:58:51.061491 13342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:58:51.061579 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:51.054063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:35091 (local address 127.3.252.126:38533)
0928 10:58:51.054295 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:51.054300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:51.054316 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:58:51.054361 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:51.054365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:51.054433 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:51.054517 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:51.055132 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.055813 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.056986 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.057415 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.057523 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.061224 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:35091: BlockingRecv error: recv got EOF from 127.3.252.67:35091 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:51.316396 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:51.307963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:44591)
0928 10:58:51.308488 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:51.308505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:51.308614 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:51.308943 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:51.308947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:51.309330 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:51.309474 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.309482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.310476 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.310486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.311450 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.311463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.312104 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.312108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.312222 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.312794 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:51.312811 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:51.316170 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":301,"threads_started":1}
W20260928 10:58:51.316551 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:51.308051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:44591 (local address 127.3.252.126:38533)
0928 10:58:51.308200 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:51.308204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:51.308568 (+ 364us) server_negotiation.cc:407] Connection header received
0928 10:58:51.308696 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:51.308699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:51.308779 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:51.308860 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:51.309598 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.310320 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.311566 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.311967 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.312054 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.316354 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:44591: BlockingRecv error: recv got EOF from 127.3.252.65:44591 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:51 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:51.720080 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:51.711190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:56711)
0928 10:58:51.711701 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:51.711717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:51.711850 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:51.712180 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:51.712184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:51.712595 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:51.712736 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.712744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.713824 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.713855 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.714968 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:51.714980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.715748 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.715753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:51.715934 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.716490 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:51.716507 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:51.719855 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":291,"threads_started":1}
W20260928 10:58:51.720189 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:51.711290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:56711 (local address 127.3.252.126:38533)
0928 10:58:51.711488 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:51.711494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:51.711803 (+ 309us) server_negotiation.cc:407] Connection header received
0928 10:58:51.711999 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:51.712003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:51.712061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:51.712139 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:51.712871 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.713664 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.715136 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:51.715591 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:51.715693 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:51.719998 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:56711: BlockingRecv error: recv got EOF from 127.3.252.66:56711 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 10:58:52 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:52 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:52.071482 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:52.062117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:52825)
0928 10:58:52.062660 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:52.062677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:52.062798 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:52.063169 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:52.063172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:52.063577 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:52.063718 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:52.063727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.064762 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.064766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:52.066031 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:52.066045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.067010 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.067016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:52.067150 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:52.067743 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:52.067761 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:52.071227 (+ 3466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
W20260928 10:58:52.071715 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:52.062235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:52825 (local address 127.3.252.126:38533)
0928 10:58:52.062384 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:52.062388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:52.062775 (+ 387us) server_negotiation.cc:407] Connection header received
0928 10:58:52.062930 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:52.062934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:52.063013 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:52.063106 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:52.063896 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.064604 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.066170 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.066797 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.066916 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:52.071509 (+ 4593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:52825: BlockingRecv error: recv got EOF from 127.3.252.67:52825 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 10:58:52 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:52 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:52.326465 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:52.317346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:52201)
0928 10:58:52.317816 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:52.317861 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:52.317983 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:52.318274 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:52.318277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:52.318766 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:52.318960 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:52.318974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.320309 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.320314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:52.321279 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:52.321293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.322177 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.322182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:52.322345 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:52.322921 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:52.322939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:52.326235 (+ 3296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":266,"threads_started":1}
W20260928 10:58:52.326710 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:52.317406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:52201 (local address 127.3.252.126:38533)
0928 10:58:52.317568 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:52.317572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:52.317950 (+ 378us) server_negotiation.cc:407] Connection header received
0928 10:58:52.318099 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:52.318103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:52.318167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:52.318243 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:52.319133 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.320177 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.321408 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.321788 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.321958 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:52.326464 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:52201: BlockingRecv error: recv got EOF from 127.3.252.65:52201 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 10:58:52 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:52 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:52.729149 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:52.720907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:57075)
0928 10:58:52.721383 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:52.721398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:52.721536 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:52.721866 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:52.721871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:52.722265 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:52.722461 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:52.722472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.723470 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.723475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:52.724397 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:52.724409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.725060 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.725066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:52.725222 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:52.725748 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:52.725764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:52.728902 (+ 3138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
W20260928 10:58:52.729257 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:52.721013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:57075 (local address 127.3.252.126:38533)
0928 10:58:52.721177 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:52.721181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:52.721481 (+ 300us) server_negotiation.cc:407] Connection header received
0928 10:58:52.721651 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:52.721655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:52.721715 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:52.721796 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:52.722599 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.723336 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.724528 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:52.724915 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:52.725030 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:52.729083 (+ 4053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:57075: BlockingRecv error: recv got EOF from 127.3.252.66:57075 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 10:58:53 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:53 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:53.081105 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:53.072365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:53415)
0928 10:58:53.072841 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:53.072860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:53.072992 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:53.073335 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:53.073339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:53.073726 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:53.073948 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:53.073959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.074929 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.074932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:53.075849 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:53.075861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.076508 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.076512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:53.076630 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:53.077209 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:53.077226 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:53.080825 (+ 3599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":268,"threads_started":1}
W20260928 10:58:53.081292 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:53.072509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:53415 (local address 127.3.252.126:38533)
0928 10:58:53.072702 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:53.072707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:53.072935 (+ 228us) server_negotiation.cc:407] Connection header received
0928 10:58:53.073136 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:53.073140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:53.073204 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:53.073291 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:53.074098 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.074822 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.075988 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.076389 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.076484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:53.081043 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:53415: BlockingRecv error: recv got EOF from 127.3.252.67:53415 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 10:58:53 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:53 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:53.335882 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:53.327345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:58871)
0928 10:58:53.327856 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:53.327935 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:53.328059 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:53.328377 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:53.328380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:53.328713 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:53.328919 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:53.328930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.329968 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.329972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:53.330874 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:53.330885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.331793 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.331799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:53.331966 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:53.332540 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:53.332558 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:53.335662 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
W20260928 10:58:53.336038 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:53.327445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:58871 (local address 127.3.252.126:38533)
0928 10:58:53.327598 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:53.327603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:53.328014 (+ 411us) server_negotiation.cc:407] Connection header received
0928 10:58:53.328189 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:53.328193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:53.328257 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:53.328350 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:53.329074 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.329805 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.331080 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.331645 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.331753 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:53.335861 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:58871: BlockingRecv error: recv got EOF from 127.3.252.65:58871 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 10:58:53 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:53 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:53.738598 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:53.730163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:57043)
0928 10:58:53.730643 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:53.730660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:53.730786 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:53.731095 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:53.731098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:53.731419 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:53.731590 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:53.731598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.732672 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.732676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:53.733655 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:53.733668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.734381 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.734387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:53.734538 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:53.735131 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:53.735154 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:53.738335 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
W20260928 10:58:53.738725 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:53.730241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:57043 (local address 127.3.252.126:38533)
0928 10:58:53.730383 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:53.730387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:53.730732 (+ 345us) server_negotiation.cc:407] Connection header received
0928 10:58:53.730923 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:53.730927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:53.730991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:53.731069 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:53.731708 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.732552 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.733784 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:53.734240 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:53.734332 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:53.738520 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:57043: BlockingRecv error: recv got EOF from 127.3.252.66:57043 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 10:58:54 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:54 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:54.090392 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:54.082032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:44929)
0928 10:58:54.082506 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:54.082522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:54.082646 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:54.082920 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:54.082924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:54.083279 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:54.083436 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:54.083445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.084457 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.084461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:54.085353 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:54.085363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.086093 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.086099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:54.086221 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:54.086755 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:54.086772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:54.090119 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":243,"threads_started":1}
W20260928 10:58:54.090543 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:54.082101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:44929 (local address 127.3.252.126:38533)
0928 10:58:54.082256 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:54.082260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:54.082630 (+ 370us) server_negotiation.cc:407] Connection header received
0928 10:58:54.082732 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:54.082736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:54.082798 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:54.082888 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:54.083592 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.084328 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.085509 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.085961 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.086062 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:54.090337 (+ 4275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:44929: BlockingRecv error: recv got EOF from 127.3.252.67:44929 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 10:58:54 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:54 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:54.345088 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:54.336776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:57049)
0928 10:58:54.337257 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:54.337273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:54.337382 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:54.337636 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:54.337639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:54.338025 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:54.338179 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:54.338187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.339189 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.339193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:54.340159 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:54.340170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.340816 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.340820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:54.340934 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:54.341474 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:54.341490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:54.344846 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
W20260928 10:58:54.345250 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:54.336849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:57049 (local address 127.3.252.126:38533)
0928 10:58:54.337051 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:54.337058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:54.337361 (+ 303us) server_negotiation.cc:407] Connection header received
0928 10:58:54.337468 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:54.337472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:54.337531 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:54.337608 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:54.338331 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.339083 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.340298 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.340707 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.340800 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:54.345039 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:57049: BlockingRecv error: recv got EOF from 127.3.252.65:57049 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 10:58:54 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:54 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:54.748608 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:54.739549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:39447)
0928 10:58:54.740250 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:54.740266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:54.740378 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:54.740652 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:54.740655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:54.741048 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:54.741205 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:54.741214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.742370 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.742376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:54.743646 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:54.743658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.744388 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.744392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:54.744524 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:54.745126 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:54.745142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:54.748368 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":495,"threads_started":1}
W20260928 10:58:54.748708 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:54.739605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:39447 (local address 127.3.252.126:38533)
0928 10:58:54.739796 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:54.739803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:54.740336 (+ 533us) server_negotiation.cc:407] Connection header received
0928 10:58:54.740466 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:54.740469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:54.740542 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:54.740625 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:54.741309 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.742224 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.743776 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:54.744226 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:54.744321 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:54.748538 (+ 4217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:39447: BlockingRecv error: recv got EOF from 127.3.252.66:39447 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 10:58:55 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:55 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:55.100142 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:55.091281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:60517)
0928 10:58:55.091796 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:55.091816 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:55.091955 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:55.092274 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:55.092278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:55.092658 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:55.092815 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:55.092835 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.093795 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.093801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:55.094911 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:55.094924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.095598 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.095604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:55.095773 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:55.096329 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:55.096347 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:55.099810 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":257,"threads_started":1}
W20260928 10:58:55.100276 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:55.091357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:60517 (local address 127.3.252.126:38533)
0928 10:58:55.091524 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:55.091530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:55.091900 (+ 370us) server_negotiation.cc:407] Connection header received
0928 10:58:55.092084 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:55.092088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:55.092148 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:55.092225 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:55.092959 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.093663 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.095051 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.095451 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.095563 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:55.100042 (+ 4479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:60517: BlockingRecv error: recv got EOF from 127.3.252.67:60517 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 10:58:55 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:55 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:55.354251 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:55.345959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:36803)
0928 10:58:55.346375 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:55.346393 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:55.346500 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:55.346783 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:55.346786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:55.347169 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:55.347319 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:55.347327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.348317 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.348321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:55.349268 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:55.349281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.349980 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.349985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:55.350115 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:55.350638 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:55.350654 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:55.354010 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
W20260928 10:58:55.354422 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:55.346008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:36803 (local address 127.3.252.126:38533)
0928 10:58:55.346208 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:55.346214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:55.346461 (+ 247us) server_negotiation.cc:407] Connection header received
0928 10:58:55.346609 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:55.346612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:55.346672 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:55.346752 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:55.347427 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.348160 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.349414 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.349798 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.349932 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:55.354175 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:36803: BlockingRecv error: recv got EOF from 127.3.252.65:36803 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 10:58:55 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:55 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:55.758111 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:55.749502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:53535)
0928 10:58:55.750069 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:55.750090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:55.750226 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:55.750512 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:55.750515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:55.750908 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:55.751049 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:55.751057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.752041 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.752046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:55.753244 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:55.753257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.753934 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.753939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:55.754059 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:55.754617 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:55.754634 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:55.757783 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":440,"threads_started":1}
W20260928 10:58:55.758232 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:55.749706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:53535 (local address 127.3.252.126:38533)
0928 10:58:55.749942 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:55.749949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:55.750184 (+ 235us) server_negotiation.cc:407] Connection header received
0928 10:58:55.750336 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:55.750340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:55.750400 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:55.750480 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:55.751174 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.751935 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.753373 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:55.753794 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:55.753951 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:55.758006 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:53535: BlockingRecv error: recv got EOF from 127.3.252.66:53535 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 10:58:56 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:56 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:56.112509 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:56.101046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:56919)
0928 10:58:56.101579 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:56.101596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:56.101724 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:56.102089 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:56.102095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:56.102505 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:56.102646 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:56.102654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.103602 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.103607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:56.104533 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:56.104544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.105220 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.105226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:56.105382 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:56.108767 (+ 3385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:56.108786 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:56.112288 (+ 3502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":277,"threads_started":1}
W20260928 10:58:56.112712 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:56.101208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:56919 (local address 127.3.252.126:38533)
0928 10:58:56.101431 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:56.101437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:56.101673 (+ 236us) server_negotiation.cc:407] Connection header received
0928 10:58:56.101853 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:56.101857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:56.101925 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:56.102038 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:56.102782 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.103477 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.104674 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.105086 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.105194 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:56.112494 (+ 7300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:56919: BlockingRecv error: recv got EOF from 127.3.252.67:56919 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 10:58:56 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:56 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:56.364289 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:56.355257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:33921)
0928 10:58:56.355741 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:56.355756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:56.355891 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:56.356190 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:56.356194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:56.356591 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:56.356757 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:56.356766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.357823 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.357849 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:56.358787 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:56.358801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.359612 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.359617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:56.359743 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:56.360281 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:56.360298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:56.363864 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":242,"threads_started":1}
W20260928 10:58:56.364321 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:56.355378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:33921 (local address 127.3.252.126:38533)
0928 10:58:56.355600 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:56.355606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:56.355841 (+ 235us) server_negotiation.cc:407] Connection header received
0928 10:58:56.356009 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:56.356013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:56.356071 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:56.356150 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:56.356909 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.357682 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.359005 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.359445 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.359558 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:56.364095 (+ 4537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:33921: BlockingRecv error: recv got EOF from 127.3.252.65:33921 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 10:58:56 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:56 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:56.767506 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:56.759233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:49009)
0928 10:58:56.759668 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:56.759683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:56.759806 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:56.760190 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:56.760195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:56.760606 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:56.760765 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:56.760794 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.761786 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.761789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:56.762751 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:56.762763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.763408 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.763412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:56.763543 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:56.764082 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:56.764101 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:56.767273 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
W20260928 10:58:56.767668 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:56.759300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:49009 (local address 127.3.252.126:38533)
0928 10:58:56.759456 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:56.759462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:56.759759 (+ 297us) server_negotiation.cc:407] Connection header received
0928 10:58:56.759966 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:56.759970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:56.760062 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:56.760140 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:56.760923 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.761672 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.762870 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:56.763272 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:56.763365 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:56.767487 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:49009: BlockingRecv error: recv got EOF from 127.3.252.66:49009 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 10:58:57 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:57 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:57.122277 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:57.113422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:42939)
0928 10:58:57.114034 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:57.114050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:57.114193 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:57.114520 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:57.114525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:57.114956 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:57.115139 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:57.115152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.116386 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.116392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:57.117368 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:57.117381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.118085 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.118090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:57.118256 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:57.118778 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:57.118794 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:57.122049 (+ 3255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":274,"threads_started":1}
W20260928 10:58:57.122426 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:57.113621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:42939 (local address 127.3.252.126:38533)
0928 10:58:57.113906 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:57.113916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:57.114140 (+ 224us) server_negotiation.cc:407] Connection header received
0928 10:58:57.114306 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:57.114312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:57.114391 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:57.114475 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:57.115286 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.116260 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.117508 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.117938 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.118040 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:57.122222 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:42939: BlockingRecv error: recv got EOF from 127.3.252.67:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 10:58:57 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:57 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:57.374280 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:57.365251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:53595)
0928 10:58:57.365819 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:57.365857 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:57.365996 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:57.366343 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:57.366348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:57.366749 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:57.366925 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:57.366935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.368245 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.368250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:57.369309 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:57.369323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.370044 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.370049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:57.370184 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:57.370782 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:57.370799 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:57.374032 (+ 3233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":288,"threads_started":1}
W20260928 10:58:57.374383 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:57.365388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:53595 (local address 127.3.252.126:38533)
0928 10:58:57.365566 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:57.365570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:57.365985 (+ 415us) server_negotiation.cc:407] Connection header received
0928 10:58:57.366101 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:57.366105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:57.366169 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:57.366272 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:57.367104 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.368083 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.369466 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.369893 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.370010 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:57.374211 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:53595: BlockingRecv error: recv got EOF from 127.3.252.65:53595 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 10:58:57 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:57 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:57.777648 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:57.768403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:37079)
0928 10:58:57.768964 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:57.768980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:57.769115 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:57.769445 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:57.769448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:57.769950 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:57.770181 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:57.770198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.771404 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.771410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:57.772371 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:57.772384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.773075 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.773080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:57.773193 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:57.773771 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:57.773794 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:57.777408 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":270,"threads_started":1}
W20260928 10:58:57.777808 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:57.768541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:37079 (local address 127.3.252.126:38533)
0928 10:58:57.768696 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:57.768700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:57.769068 (+ 368us) server_negotiation.cc:407] Connection header received
0928 10:58:57.769234 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:57.769239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:57.769320 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:57.769429 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:57.770317 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.771245 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.772494 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:57.772907 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:57.773029 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:57.777596 (+ 4567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:37079: BlockingRecv error: recv got EOF from 127.3.252.66:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 10:58:58 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:58 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:58.133313 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:58.123224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:58083)
0928 10:58:58.123819 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:58.123840 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:58.124025 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:58.124422 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:58.124427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:58.124874 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:58.125100 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:58.125113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.126325 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.126331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:58.128057 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:58.128073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.128889 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.128894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:58.129012 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:58.129532 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:58.129550 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:58.132945 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":283,"threads_started":1}
W20260928 10:58:58.133364 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:58.123253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:58083 (local address 127.3.252.126:38533)
0928 10:58:58.123455 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:58.123461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:58.123952 (+ 491us) server_negotiation.cc:407] Connection header received
0928 10:58:58.124167 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:58.124173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:58.124268 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:58.124370 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:58.125290 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.126180 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.128194 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.128731 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.128829 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:58.133150 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:58083: BlockingRecv error: recv got EOF from 127.3.252.67:58083 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 10:58:58 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:58 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:58.383695 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:58.375236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:57937)
0928 10:58:58.375748 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:58.375764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:58.375877 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:58.376172 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:58.376174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:58.376549 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:58.376704 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:58.376712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.377694 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.377698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:58.378664 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:58.378676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.379365 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.379371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:58.379498 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:58.380071 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:58.380088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:58.383465 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
W20260928 10:58:58.383865 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:58.375406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:57937 (local address 127.3.252.126:38533)
0928 10:58:58.375566 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:58.375571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:58.375829 (+ 258us) server_negotiation.cc:407] Connection header received
0928 10:58:58.375971 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:58.375974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:58.376032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:58.376134 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:58.376814 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.377568 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.378783 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.379192 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.379281 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:58.383656 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:57937: BlockingRecv error: recv got EOF from 127.3.252.65:57937 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 10:58:58 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:58 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:58.788287 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:58.778850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:37815)
0928 10:58:58.779487 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:58.779506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:58.779645 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:58.780031 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:58.780035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:58.780478 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:58.780643 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:58.780653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.782176 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.782180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:58.783242 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:58.783255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.784009 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.784015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:58.784136 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:58.784743 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:58.784761 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:58.788040 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":323,"threads_started":1}
W20260928 10:58:58.788643 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:58.778987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:37815 (local address 127.3.252.126:38533)
0928 10:58:58.779173 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:58.779179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:58.779578 (+ 399us) server_negotiation.cc:407] Connection header received
0928 10:58:58.779772 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:58.779778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:58.779875 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:58.779973 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:58.780773 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.782024 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.783371 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:58.783782 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:58.783897 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:58.788411 (+ 4514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:37815: BlockingRecv error: recv got EOF from 127.3.252.66:37815 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 10:58:59 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:59 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:59.143081 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:59.134487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:58509)
0928 10:58:59.135084 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:59.135102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:59.135253 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:59.135575 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:59.135580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:59.135996 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:59.136156 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:59.136165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.137126 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.137131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:59.138070 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:59.138081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.138768 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.138774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:59.138923 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:59.139475 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:59.139492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:59.142730 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":283,"threads_started":1}
W20260928 10:58:59.143167 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:59.134591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:58509 (local address 127.3.252.126:38533)
0928 10:58:59.134795 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:59.134802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:59.135165 (+ 363us) server_negotiation.cc:407] Connection header received
0928 10:58:59.135379 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:59.135382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:59.135448 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:59.135526 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:59.136292 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.136998 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.138198 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.138602 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.138711 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:59.142948 (+ 4237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:58509: BlockingRecv error: recv got EOF from 127.3.252.67:58509 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 10:58:59 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:59 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:59.393393 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:59.384729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:41803)
0928 10:58:59.385251 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:59.385268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:59.385374 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:59.385733 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:59.385738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:59.386227 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:59.386387 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:59.386400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.387507 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.387511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:59.388432 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:59.388444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.389236 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.389241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:59.389369 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:59.389972 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:59.390013 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:59.393171 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":286,"threads_started":1}
W20260928 10:58:59.393558 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:59.384819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:41803 (local address 127.3.252.126:38533)
0928 10:58:59.385039 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:59.385046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:59.385331 (+ 285us) server_negotiation.cc:407] Connection header received
0928 10:58:59.385460 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:59.385464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:59.385561 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:59.385645 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:59.386552 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.387367 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.388574 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.389044 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.389138 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:59.393365 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:41803: BlockingRecv error: recv got EOF from 127.3.252.65:41803 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 10:58:59 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:58:59 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:58:59.798064 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:58:59.789303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:35323)
0928 10:58:59.789804 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 10:58:59.789821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:58:59.789975 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:58:59.790297 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:58:59.790301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:58:59.790665 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:58:59.790809 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:59.790818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.791898 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.791902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:59.792871 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:58:59.792883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.793543 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.793547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:58:59.793666 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:59.794289 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:58:59.794311 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:58:59.797772 (+ 3461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
W20260928 10:58:59.798239 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:58:59.789344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:35323 (local address 127.3.252.126:38533)
0928 10:58:59.789495 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 10:58:59.789499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:58:59.789918 (+ 419us) server_negotiation.cc:407] Connection header received
0928 10:58:59.790090 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:58:59.790096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:58:59.790174 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:58:59.790256 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:58:59.790971 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.791758 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.793005 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:58:59.793403 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:58:59.793504 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:58:59.798026 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:35323: BlockingRecv error: recv got EOF from 127.3.252.66:35323 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 10:59:00 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:00 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:00.152483 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:00.144041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:55741)
0928 10:59:00.144509 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:00.144525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:00.144630 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:00.144912 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:00.144915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:00.145307 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:00.145445 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:00.145454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.146566 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.146569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:00.147516 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:00.147527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.148221 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.148225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:00.148351 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:00.148927 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:00.148944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:00.152261 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
W20260928 10:59:00.152761 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:00.144103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:55741 (local address 127.3.252.126:38533)
0928 10:59:00.144257 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:00.144261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:00.144584 (+ 323us) server_negotiation.cc:407] Connection header received
0928 10:59:00.144722 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:00.144726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:00.144787 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:00.144872 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:00.145601 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.146448 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.147652 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.148083 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.148178 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:00.152473 (+ 4295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:55741: BlockingRecv error: recv got EOF from 127.3.252.67:55741 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 10:59:00 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:00 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:00.403827 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:00.394402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:53251)
0928 10:59:00.394875 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:00.394892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:00.395016 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:00.395296 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:00.395300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:00.395675 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:00.395832 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:00.395841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.396836 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.396840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:00.397884 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:00.397899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.398709 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.398715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:00.398839 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:00.399546 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:00.399566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:00.403586 (+ 4020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":234,"threads_started":1}
W20260928 10:59:00.404136 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:00.394571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:53251 (local address 127.3.252.126:38533)
0928 10:59:00.394830 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:00.394837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:00.395007 (+ 170us) server_negotiation.cc:407] Connection header received
0928 10:59:00.395109 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:00.395113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:00.395177 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:00.395268 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:00.395962 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.396702 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.398074 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.398529 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.398641 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:00.403849 (+ 5208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:53251: BlockingRecv error: recv got EOF from 127.3.252.65:53251 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 28 10:59:00 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:00 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:00.807300 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:00.798968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:37921)
0928 10:59:00.799474 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:00.799494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:00.799643 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:00.799963 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:00.799968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:00.800411 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:00.800622 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:00.800635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.801764 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.801768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:00.802771 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:00.802783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.803366 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.803369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:00.803471 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:00.804056 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:00.804073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:00.807038 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
W20260928 10:59:00.807448 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:00.799048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:37921 (local address 127.3.252.126:38533)
0928 10:59:00.799260 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:00.799266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:00.799607 (+ 341us) server_negotiation.cc:407] Connection header received
0928 10:59:00.799760 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:00.799765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:00.799831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:00.799907 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:00.800767 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.801640 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.802893 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:00.803258 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:00.803346 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:00.807207 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:37921: BlockingRecv error: recv got EOF from 127.3.252.66:37921 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 10:59:01 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:01 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:01.165762 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:01.153372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:45161)
0928 10:59:01.153817 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:01.153854 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:01.153962 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:01.154252 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:01.154255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:01.154617 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:01.154771 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:01.154779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.155714 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.155718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:01.156742 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:01.156757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.157669 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.157676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:01.157860 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:01.162496 (+ 4636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:01.162514 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:01.165525 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
W20260928 10:59:01.165956 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:01.153535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:45161 (local address 127.3.252.126:38533)
0928 10:59:01.153789 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:01.153794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:01.153912 (+ 118us) server_negotiation.cc:407] Connection header received
0928 10:59:01.154069 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:01.154072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:01.154147 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:01.154225 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:01.154877 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.155614 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.156898 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.157519 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.157653 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:01.165710 (+ 8057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:45161: BlockingRecv error: recv got EOF from 127.3.252.67:45161 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 28 10:59:01 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:01 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:01.413321 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:01.404725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:49339)
0928 10:59:01.405227 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:01.405246 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:01.405383 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:01.405703 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:01.405708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:01.406225 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:01.406445 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:01.406458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.407441 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.407446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:01.408426 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:01.408439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.409298 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.409305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:01.409426 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:01.410007 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:01.410025 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:01.413099 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":293,"threads_started":1}
W20260928 10:59:01.413443 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:01.404839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:49339 (local address 127.3.252.126:38533)
0928 10:59:01.405021 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:01.405027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:01.405314 (+ 287us) server_negotiation.cc:407] Connection header received
0928 10:59:01.405496 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:01.405501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:01.405569 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:01.405645 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:01.406580 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.407289 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.408558 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.409153 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.409245 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:01.413275 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:49339: BlockingRecv error: recv got EOF from 127.3.252.65:49339 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 10:59:01 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:01 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:01.816793 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:01.808256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:58285)
0928 10:59:01.808688 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:01.808704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:01.808812 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:01.809078 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:01.809081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:01.809491 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:01.809663 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:01.809672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.810785 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.810790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:01.811724 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:01.811735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.812400 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.812405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:01.812539 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:01.813076 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:01.813093 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:01.816523 (+ 3430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":230,"threads_started":1}
W20260928 10:59:01.816946 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:01.808352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:58285 (local address 127.3.252.126:38533)
0928 10:59:01.808510 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:01.808515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:01.808768 (+ 253us) server_negotiation.cc:407] Connection header received
0928 10:59:01.808900 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:01.808904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:01.808965 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:01.809043 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:01.809794 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.810648 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.811847 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:01.812243 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:01.812334 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:01.816682 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:58285: BlockingRecv error: recv got EOF from 127.3.252.66:58285 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 10:59:02 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:02 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:02.174763 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:02.166639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:35803)
0928 10:59:02.167082 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:02.167098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:02.167210 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:02.167495 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:02.167498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:02.167872 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:02.168048 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:02.168057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.169042 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.169046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:02.170064 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:02.170085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.170737 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.170741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:02.170862 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:02.171384 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:02.171401 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:02.174548 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":245,"threads_started":1}
W20260928 10:59:02.174943 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:02.166737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:35803 (local address 127.3.252.126:38533)
0928 10:59:02.166931 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:02.166937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:02.167190 (+ 253us) server_negotiation.cc:407] Connection header received
0928 10:59:02.167298 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:02.167301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:02.167371 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:02.167454 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:02.168156 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.168921 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.170228 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.170622 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.170727 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:02.174731 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:35803: BlockingRecv error: recv got EOF from 127.3.252.67:35803 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 10:59:02 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:02 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:02.423295 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:02.414232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:33101)
0928 10:59:02.414719 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:02.414737 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:02.414872 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:02.415225 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:02.415228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:02.415641 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:02.415857 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:02.415868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.417033 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.417038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:02.418082 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:02.418095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.418855 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.418861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:02.419019 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:02.419539 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:02.419557 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:02.423027 (+ 3470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
W20260928 10:59:02.423660 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:02.414296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:33101 (local address 127.3.252.126:38533)
0928 10:59:02.414543 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:02.414548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:02.414811 (+ 263us) server_negotiation.cc:407] Connection header received
0928 10:59:02.415026 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:02.415031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:02.415100 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:02.415182 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:02.416007 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.416900 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.418214 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.418692 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.418797 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:02.423319 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:33101: BlockingRecv error: recv got EOF from 127.3.252.65:33101 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 10:59:02 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:02 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:02.826321 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:02.817867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:52655)
0928 10:59:02.818326 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:02.818341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:02.818459 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:02.818773 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:02.818776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:02.819160 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:02.819322 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:02.819331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.820399 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.820403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:02.821450 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:02.821468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.822323 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.822327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:02.822451 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:02.823038 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:02.823055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:02.826072 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
W20260928 10:59:02.826463 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:02.817916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:52655 (local address 127.3.252.126:38533)
0928 10:59:02.818119 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:02.818125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:02.818412 (+ 287us) server_negotiation.cc:407] Connection header received
0928 10:59:02.818584 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:02.818587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:02.818648 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:02.818724 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:02.819472 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.820261 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.821611 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:02.822151 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:02.822264 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:02.826244 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:52655: BlockingRecv error: recv got EOF from 127.3.252.66:52655 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 10:59:03 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:03 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:03.184315 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:03.175643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:42137)
0928 10:59:03.176212 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:03.176229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:03.176372 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:03.176693 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:03.176699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:03.177094 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:03.177253 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:03.177262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.178336 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.178341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:03.179266 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:03.179278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.180068 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.180075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:03.180235 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:03.180817 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:03.180835 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:03.184046 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":298,"threads_started":1}
W20260928 10:59:03.184473 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:03.175769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:42137 (local address 127.3.252.126:38533)
0928 10:59:03.175993 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:03.176000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:03.176327 (+ 327us) server_negotiation.cc:407] Connection header received
0928 10:59:03.176487 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:03.176492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:03.176562 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:03.176644 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:03.177392 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.178208 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.179406 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.179900 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.180008 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:03.184261 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:42137: BlockingRecv error: recv got EOF from 127.3.252.67:42137 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 10:59:03 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:03 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:03.432578 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:03.424201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:34023)
0928 10:59:03.424656 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:03.424672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:03.424795 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:03.425092 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:03.425097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:03.425499 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:03.425647 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:03.425655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.426627 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.426638 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:03.427531 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:03.427543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.428184 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.428190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:03.428346 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:03.428869 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:03.428885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:03.432327 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
W20260928 10:59:03.432781 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:03.424298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:34023 (local address 127.3.252.126:38533)
0928 10:59:03.424472 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:03.424477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:03.424748 (+ 271us) server_negotiation.cc:407] Connection header received
0928 10:59:03.424903 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:03.424907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:03.424966 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:03.425044 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:03.425787 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.426494 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.427669 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.428053 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.428167 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:03.432519 (+ 4352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:34023: BlockingRecv error: recv got EOF from 127.3.252.65:34023 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 10:59:03 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:03 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:03.835604 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:03.827204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:47895)
0928 10:59:03.827648 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:03.827663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:03.827785 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:03.828070 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:03.828074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:03.828506 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:03.828655 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:03.828665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.829621 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.829627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:03.830543 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:03.830555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.831231 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.831237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:03.831396 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:03.831955 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:03.831971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:03.835299 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
W20260928 10:59:03.835668 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:03.827370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:47895 (local address 127.3.252.126:38533)
0928 10:59:03.827572 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:03.827578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:03.827730 (+ 152us) server_negotiation.cc:407] Connection header received
0928 10:59:03.827884 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:03.827889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:03.827951 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:03.828029 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:03.828785 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.829490 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.830682 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:03.831097 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:03.831206 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:03.835466 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:47895: BlockingRecv error: recv got EOF from 127.3.252.66:47895 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 10:59:04 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:04 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.67@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:04.193708 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:04.185299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.67:60607)
0928 10:59:04.185719 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:04.185736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:04.185872 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:04.186128 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:04.186130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:04.186511 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:04.186649 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:04.186656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.187589 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.187592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:04.188606 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:04.188619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.189491 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.189496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:04.189646 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:04.190267 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:04.190284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:04.193402 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
W20260928 10:59:04.193856 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:04.185365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:60607 (local address 127.3.252.126:38533)
0928 10:59:04.185573 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:04.185580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:04.185809 (+ 229us) server_negotiation.cc:407] Connection header received
0928 10:59:04.185958 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:04.185962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:04.186022 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:04.186101 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:04.186768 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.187478 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.188714 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.189374 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.189488 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:04.193614 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.67:60607: BlockingRecv error: recv got EOF from 127.3.252.67:60607 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 10:59:04 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:04 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.65@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:04.442555 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:04.433931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.65:54847)
0928 10:59:04.434411 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:04.434427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:04.434577 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:04.434888 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:04.434893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:04.435282 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:04.435453 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:04.435469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.436432 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.436438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:04.437345 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:04.437357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.438117 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.438122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:04.438291 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:04.438847 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:04.438864 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:04.442208 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":222,"threads_started":1}
W20260928 10:59:04.442675 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:04.434000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:54847 (local address 127.3.252.126:38533)
0928 10:59:04.434301 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:04.434307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:04.434509 (+ 202us) server_negotiation.cc:407] Connection header received
0928 10:59:04.434702 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:04.434707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:04.434767 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:04.434844 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:04.435606 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.436296 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.437507 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.437984 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.438092 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:04.442426 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.65:54847: BlockingRecv error: recv got EOF from 127.3.252.65:54847 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 28 10:59:04 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:59:04 dist-test-slave-djjw krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.252.66@KRBTEST.COM for oryx/127.3.252.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:59:04.845238 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:04.837041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:38533 (local address 127.3.252.66:33219)
0928 10:59:04.837702 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:04.837721 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:04.837854 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:04.838131 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:04.838134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:04.838484 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:04.838658 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:04.838668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.839622 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.839626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:04.840526 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:04.840537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.841117 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.841121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:04.841238 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:04.841852 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:04.841870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:04.844987 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.252.126:38533: Server oryx/127.3.252.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":417,"threads_started":1}
W20260928 10:59:04.845360 13345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:04.837120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:33219 (local address 127.3.252.126:38533)
0928 10:59:04.837323 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:04.837330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:04.837793 (+ 463us) server_negotiation.cc:407] Connection header received
0928 10:59:04.837941 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:04.837944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:04.838003 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:04.838079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:04.838765 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.839482 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.840632 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.841007 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.841098 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:04.845173 (+ 4075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.252.66:33219: BlockingRecv error: recv got EOF from 127.3.252.66:33219 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:59:04.965075 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12929
I20260928 10:59:04.976733 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:04.966191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.3.252.65:41081)
0928 10:59:04.966703 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:04.966719 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:04.966747 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:59:04.966814 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:04.966821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:04.967042 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:04.967210 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:04.968551 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.969486 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.970721 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.971266 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.971368 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:04.973914 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:04.973922 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:04.973936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:04.973979 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:04.976242 (+ 2263us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.252.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:59:04.976517 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45456: Request ticket server kudu/127.3.252.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
W20260928 10:59:04.976825 13391 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45456: Request ticket server kudu/127.3.252.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:59:04.976910 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12929: Not authorized: Client connection negotiation failed: client connection to 127.3.252.65:41081: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.252.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:59:04.977125 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 12929
I20260928 10:59:04.991096 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13080
I20260928 10:59:05.003590 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:04.992187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.3.252.66:46453)
0928 10:59:04.992740 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:04.992747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:04.992774 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:59:04.992831 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:04.992838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:04.993066 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:04.993233 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:04.994833 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.995874 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.997043 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:04.997604 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:04.997702 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:05.000346 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:05.000355 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:05.000370 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:05.000436 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:05.003020 (+ 2584us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.252.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:59:05.003327 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36156: Request ticket server kudu/127.3.252.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":240,"threads_started":1}
W20260928 10:59:05.003690 13392 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36156: Request ticket server kudu/127.3.252.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:59:05.003712 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13080: Not authorized: Client connection negotiation failed: client connection to 127.3.252.66:46453: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.252.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:59:05.003862 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13080
I20260928 10:59:05.017271 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13213
I20260928 10:59:05.029300 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:05.018422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.3.252.67:42703)
0928 10:59:05.019006 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:05.019013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:05.019039 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:59:05.019101 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:05.019107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:05.019377 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:05.019571 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:05.021025 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:05.022096 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:05.023296 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:05.023824 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:05.023939 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:05.026348 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:05.026357 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:05.026373 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:05.026406 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:05.028634 (+ 2228us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.252.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:59:05.028975 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49122: Request ticket server kudu/127.3.252.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
W20260928 10:59:05.029343 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13213: Not authorized: Client connection negotiation failed: client connection to 127.3.252.67:42703: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.252.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:59:05.029397 13393 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49122: Request ticket server kudu/127.3.252.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:59:05.029500 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13213
I20260928 10:59:05.043141 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 12860
I20260928 10:59:05.044549 12922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:05.156476 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 12860
2026-09-28T10:59:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20791 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T10:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:59:05Z Disabled control of system clock
I20260928 10:59:05.225940 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40359
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:40359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 10:59:05.562141 13398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:05.562427 13398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:05.562528 13398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:05.569684 13398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:59:05.569788 13398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:05.569844 13398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:05.569911 13398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:59:05.569967 13398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:59:05.579192 13398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:40359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:05.580533 13398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:05.582412 13398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:05.593997 13403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:05.593911 13406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:05.593948 13404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:05.594286 13398 server_base.cc:1034] running on GCE node
I20260928 10:59:05.595329 13398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:05.596743 13398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:05.598012 13398 hybrid_clock.cc:648] HybridClock initialized: now 1790593145597963 us; error 32 us; skew 500 ppm
I20260928 10:59:05.598495 13398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:05.600824 13398 webserver.cc:460] Webserver started at http://127.3.252.126:37349/ using document root <none> and password file <none>
I20260928 10:59:05.601533 13398 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:05.601636 13398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:05.601997 13398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:05.604883 13398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "b2493ae12aac4632ae7a865f5c5178a9"
format_stamp: "Formatted at 2026-09-28 10:59:05 on dist-test-slave-djjw"
server_key: "d31edf4d99788224ff0b26481a93ddfc"
server_key_iv: "3a6f7b21260bc4625a6bfaa3c0cf4d16"
server_key_version: "encryptionkey@0"
I20260928 10:59:05.605641 13398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "b2493ae12aac4632ae7a865f5c5178a9"
format_stamp: "Formatted at 2026-09-28 10:59:05 on dist-test-slave-djjw"
server_key: "d31edf4d99788224ff0b26481a93ddfc"
server_key_iv: "3a6f7b21260bc4625a6bfaa3c0cf4d16"
server_key_version: "encryptionkey@0"
I20260928 10:59:05.610682 13398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:59:05.613968 13412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:05.615226 13398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:59:05.615379 13398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "b2493ae12aac4632ae7a865f5c5178a9"
format_stamp: "Formatted at 2026-09-28 10:59:05 on dist-test-slave-djjw"
server_key: "d31edf4d99788224ff0b26481a93ddfc"
server_key_iv: "3a6f7b21260bc4625a6bfaa3c0cf4d16"
server_key_version: "encryptionkey@0"
I20260928 10:59:05.615567 13398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:05.636751 13398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:05.637609 13398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:05.637905 13398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:05.654036 13398 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:40359
I20260928 10:59:05.654062 13463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:40359 every 8 connection(s)
I20260928 10:59:05.655248 13398 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:59:05.655934 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13398
I20260928 10:59:05.656131 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:59:05.656790 4081 external_mini_cluster.cc:1428] Setting key f934f567b352a80ed5210c6230b9f7d6
I20260928 10:59:05.659343 13464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:05.671943 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:05.658804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.3.252.126:40359)
0928 10:59:05.659820 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:05.659834 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:05.659884 (+ 50us) server_negotiation.cc:407] Connection header received
0928 10:59:05.661990 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:05.662018 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:05.662615 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:05.663050 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:05.663560 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:05.665511 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:05.666914 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:05.667869 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:05.668049 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:05.669137 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:05.669150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:05.669169 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:05.669917 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:05.670010 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:05.670040 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:05.670164 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:05.670398 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 10:59:05.670722 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":318,"threads_started":1}
I20260928 10:59:05.672581 13464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9: Bootstrap starting.
I20260928 10:59:05.675979 13464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:05.676923 13464 log.cc:824] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:05.679473 13464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9: No bootstrap required, opened a new log
I20260928 10:59:05.686689 13464 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2493ae12aac4632ae7a865f5c5178a9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40359 } }
I20260928 10:59:05.687052 13464 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:05.687135 13464 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2493ae12aac4632ae7a865f5c5178a9, State: Initialized, Role: FOLLOWER
I20260928 10:59:05.687654 13464 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2493ae12aac4632ae7a865f5c5178a9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40359 } }
I20260928 10:59:05.687851 13464 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:59:05.687971 13464 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:59:05.688122 13464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:05.690227 13464 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2493ae12aac4632ae7a865f5c5178a9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40359 } }
I20260928 10:59:05.690716 13464 leader_election.cc:302] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2493ae12aac4632ae7a865f5c5178a9; no voters:
I20260928 10:59:05.691247 13464 leader_election.cc:288] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:59:05.691381 13469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:59:05.692008 13469 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [term 1 LEADER]: Becoming Leader. State: Replica: b2493ae12aac4632ae7a865f5c5178a9, State: Running, Role: LEADER
I20260928 10:59:05.692536 13469 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2493ae12aac4632ae7a865f5c5178a9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40359 } }
I20260928 10:59:05.693089 13464 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:59:05.695348 13471 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2493ae12aac4632ae7a865f5c5178a9. Latest consensus state: current_term: 1 leader_uuid: "b2493ae12aac4632ae7a865f5c5178a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2493ae12aac4632ae7a865f5c5178a9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40359 } } }
I20260928 10:59:05.695741 13471 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:59:05.697523 13478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:59:05.697431 13470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2493ae12aac4632ae7a865f5c5178a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2493ae12aac4632ae7a865f5c5178a9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40359 } } }
I20260928 10:59:05.697798 13470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:59:05.701862 13478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:59:05.709784 13478 catalog_manager.cc:1347] Generated new cluster ID: 6a9228d97aaa440aa6061bd5b9ef3070
I20260928 10:59:05.709906 13478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:59:05.725775 13478 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:59:05.727016 13478 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:59:05.736275 13478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2493ae12aac4632ae7a865f5c5178a9: Generated new TSK 0
I20260928 10:59:05.737056 13478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:59:05.744578 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40359
--builtin_ntp_servers=127.3.252.84:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 10:59:06.083837 13488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:06.084139 13488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:06.084259 13488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:06.091526 13488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:06.091648 13488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:06.091785 13488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:59:06.101218 13488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:06.102638 13488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:06.104406 13488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:06.117538 13493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:06.117636 13494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:06.117570 13496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:06.118233 13488 server_base.cc:1034] running on GCE node
I20260928 10:59:06.118868 13488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:06.122212 13488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:06.123450 13488 hybrid_clock.cc:648] HybridClock initialized: now 1790593146123361 us; error 66 us; skew 500 ppm
I20260928 10:59:06.123888 13488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:06.126762 13488 webserver.cc:460] Webserver started at http://127.3.252.65:38909/ using document root <none> and password file <none>
I20260928 10:59:06.127689 13488 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:06.127787 13488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:06.128078 13488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:06.130923 13488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "2c44033968a7470da94d39fe3b6f60e4"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "2feb1a6d11fff9f8ae65a80522aa9a28"
server_key_iv: "f8348ce20f37bb4ab0bebc04e470656a"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.131668 13488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "2c44033968a7470da94d39fe3b6f60e4"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "2feb1a6d11fff9f8ae65a80522aa9a28"
server_key_iv: "f8348ce20f37bb4ab0bebc04e470656a"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.136535 13488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:59:06.139796 13502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:06.141161 13488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:59:06.141326 13488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "2c44033968a7470da94d39fe3b6f60e4"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "2feb1a6d11fff9f8ae65a80522aa9a28"
server_key_iv: "f8348ce20f37bb4ab0bebc04e470656a"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.141515 13488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:06.160602 13488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:06.161461 13488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:06.161741 13488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:06.162694 13488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:06.164122 13488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:06.164214 13488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:06.164319 13488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:06.164402 13488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:06.193502 13488 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:44519
I20260928 10:59:06.193588 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:44519 every 8 connection(s)
I20260928 10:59:06.194779 13488 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:59:06.195123 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13488
I20260928 10:59:06.195315 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:59:06.196121 4081 external_mini_cluster.cc:1428] Setting key 05c130473bd5d3d2844f822f0880b002
I20260928 10:59:06.199329 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40359
--builtin_ntp_servers=127.3.252.84:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 10:59:06.215502 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:06.197587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:55547 (local address 127.3.252.126:40359)
0928 10:59:06.197995 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:06.198001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:06.199413 (+ 1412us) server_negotiation.cc:407] Connection header received
0928 10:59:06.201756 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:06.201761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:06.201860 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:06.201957 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:06.203219 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.203891 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.205943 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.206357 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.206450 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:06.214555 (+ 8105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:06.214562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:06.214566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:06.214594 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:06.214650 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:06.214668 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:06.214781 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:06.215012 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 10:59:06.215131 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":189,"threads_started":1}
I20260928 10:59:06.216467 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:06.198167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40359 (local address 127.3.252.65:55547)
0928 10:59:06.199240 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:06.199297 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:06.201470 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:06.202210 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:06.202221 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:06.202333 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:06.203070 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:06.203094 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.204039 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.204045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:06.205796 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:06.205810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.206562 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.206568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:06.206855 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:06.214095 (+ 7240us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:59:06.214098 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:06.214153 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:59:06.214275 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:06.214792 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:06.214801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:06.214809 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:59:06.214959 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 10:59:06.215353 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":276,"threads_started":1}
I20260928 10:59:06.218072 13615 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40359
I20260928 10:59:06.218390 13615 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:06.219225 13615 heartbeater.cc:511] Master 127.3.252.126:40359 requested a full tablet report, sending...
I20260928 10:59:06.221143 13429 ts_manager.cc:194] Registered new tserver with Master: 2c44033968a7470da94d39fe3b6f60e4 (127.3.252.65:44519)
I20260928 10:59:06.222640 13429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.65:55547
W20260928 10:59:06.517393 13619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:06.517678 13619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:06.517782 13619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:06.525177 13619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:06.525282 13619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:06.525418 13619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:59:06.534550 13619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:06.535827 13619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:06.537525 13619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:06.549808 13625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:06.549876 13624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:06.550064 13627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:06.550726 13619 server_base.cc:1034] running on GCE node
I20260928 10:59:06.551309 13619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:06.552222 13619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:06.553431 13619 hybrid_clock.cc:648] HybridClock initialized: now 1790593146553371 us; error 56 us; skew 500 ppm
I20260928 10:59:06.553747 13619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:06.556341 13619 webserver.cc:460] Webserver started at http://127.3.252.66:37817/ using document root <none> and password file <none>
I20260928 10:59:06.557060 13619 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:06.557163 13619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:06.557480 13619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:06.560333 13619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "2fab7026942c4997b7d9a9ae8de3a20f"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "d1b9949d6cc5f4705e7c4c1ee3fd6618"
server_key_iv: "8d1dce8d862a0cc706826508afcc3186"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.561081 13619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "2fab7026942c4997b7d9a9ae8de3a20f"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "d1b9949d6cc5f4705e7c4c1ee3fd6618"
server_key_iv: "8d1dce8d862a0cc706826508afcc3186"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.566303 13619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 10:59:06.569990 13633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:06.571466 13619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:59:06.571651 13619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "2fab7026942c4997b7d9a9ae8de3a20f"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "d1b9949d6cc5f4705e7c4c1ee3fd6618"
server_key_iv: "8d1dce8d862a0cc706826508afcc3186"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.571861 13619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:06.593032 13619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:06.593937 13619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:06.594214 13619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:06.595139 13619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:06.596589 13619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:06.596684 13619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:06.596793 13619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:06.596858 13619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:06.625918 13619 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:39841
I20260928 10:59:06.625937 13745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:39841 every 8 connection(s)
I20260928 10:59:06.627133 13619 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:59:06.631954 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13619
I20260928 10:59:06.632099 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:59:06.632532 4081 external_mini_cluster.cc:1428] Setting key fb93beb746efde5a74566634c9d74c32
I20260928 10:59:06.635156 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40359
--builtin_ntp_servers=127.3.252.84:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 10:59:06.639357 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:06.629521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:55545 (local address 127.3.252.126:40359)
0928 10:59:06.629772 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:06.629778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:06.631085 (+ 1307us) server_negotiation.cc:407] Connection header received
0928 10:59:06.632759 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:06.632765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:06.632843 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:06.632927 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:06.634271 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.635288 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.636814 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.637306 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.637419 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:06.638621 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:06.638627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:06.638631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:06.638651 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:06.638687 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:06.638700 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:06.638780 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:06.638950 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 10:59:06.639037 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 10:59:06.640280 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:06.630024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40359 (local address 127.3.252.66:55545)
0928 10:59:06.630896 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:06.630930 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:06.632500 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:06.633153 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:06.633164 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:06.633271 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:06.633884 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:06.633907 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.635471 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.635476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:06.636681 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:06.636692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:06.637485 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:06.637490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:06.637685 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:06.638339 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:59:06.638341 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:06.638379 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:59:06.638458 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:06.638782 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:06.638787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:06.638792 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 10:59:06.638915 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 10:59:06.639274 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":263,"threads_started":1}
I20260928 10:59:06.641543 13746 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40359
I20260928 10:59:06.641809 13746 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:06.642544 13746 heartbeater.cc:511] Master 127.3.252.126:40359 requested a full tablet report, sending...
I20260928 10:59:06.643800 13429 ts_manager.cc:194] Registered new tserver with Master: 2fab7026942c4997b7d9a9ae8de3a20f (127.3.252.66:39841)
I20260928 10:59:06.644637 13429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.66:55545
W20260928 10:59:06.947839 13749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:06.948114 13749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:06.948216 13749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:06.955279 13749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:06.955385 13749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:06.955514 13749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:59:06.964490 13749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:06.965812 13749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:06.967581 13749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:06.980230 13755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:06.980249 13757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:06.980815 13749 server_base.cc:1034] running on GCE node
W20260928 10:59:06.980924 13754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:06.981561 13749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:06.982503 13749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:06.983691 13749 hybrid_clock.cc:648] HybridClock initialized: now 1790593146983646 us; error 41 us; skew 500 ppm
I20260928 10:59:06.983973 13749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:06.986184 13749 webserver.cc:460] Webserver started at http://127.3.252.67:42747/ using document root <none> and password file <none>
I20260928 10:59:06.986835 13749 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:06.986920 13749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:06.987179 13749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:06.989934 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "e4943021ca9a48e2825025b3307e8d49"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "d045e11563f7fa529260e3a234b27aa4"
server_key_iv: "440ed4b994077cd625e1c16a02e0dbb8"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.990626 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "e4943021ca9a48e2825025b3307e8d49"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "d045e11563f7fa529260e3a234b27aa4"
server_key_iv: "440ed4b994077cd625e1c16a02e0dbb8"
server_key_version: "encryptionkey@0"
I20260928 10:59:06.995584 13749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:59:06.999042 13763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:07.000277 13749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:59:07.000425 13749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "e4943021ca9a48e2825025b3307e8d49"
format_stamp: "Formatted at 2026-09-28 10:59:06 on dist-test-slave-djjw"
server_key: "d045e11563f7fa529260e3a234b27aa4"
server_key_iv: "440ed4b994077cd625e1c16a02e0dbb8"
server_key_version: "encryptionkey@0"
I20260928 10:59:07.000614 13749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:07.018762 13749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:07.019564 13749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:07.019866 13749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:07.020772 13749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:07.022149 13749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:07.022244 13749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:07.022344 13749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:07.022411 13749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:07.051141 13749 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:44541
I20260928 10:59:07.051188 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:44541 every 8 connection(s)
I20260928 10:59:07.052361 13749 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:59:07.055958 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13749
I20260928 10:59:07.056114 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:59:07.056607 4081 external_mini_cluster.cc:1428] Setting key fa6fcb3f49ddd078b84ac9881e98508e
I20260928 10:59:07.064312 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:07.054749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:44143 (local address 127.3.252.126:40359)
0928 10:59:07.054944 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:07.054950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:07.056289 (+ 1339us) server_negotiation.cc:407] Connection header received
0928 10:59:07.058225 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:07.058230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:07.058334 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:07.058439 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:07.059719 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.060433 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.061927 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.062384 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.062480 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:07.063652 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:07.063657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:07.063660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:07.063676 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:07.063707 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:07.063719 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:07.063785 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:07.063988 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 10:59:07.064075 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:59:07.065290 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:07.055258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40359 (local address 127.3.252.67:44143)
0928 10:59:07.056120 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:07.056163 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:07.057983 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:07.058672 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:07.058683 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:07.058799 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:07.059570 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:07.059590 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.060553 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.060557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:07.061759 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:07.061775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.062558 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.062562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:07.062766 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:07.063370 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:59:07.063372 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:07.063412 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:59:07.063487 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:07.063790 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:07.063796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:07.063802 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:59:07.063937 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 10:59:07.064271 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":265,"threads_started":1}
I20260928 10:59:07.066665 13876 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40359
I20260928 10:59:07.066944 13876 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:07.067672 13876 heartbeater.cc:511] Master 127.3.252.126:40359 requested a full tablet report, sending...
I20260928 10:59:07.068996 13429 ts_manager.cc:194] Registered new tserver with Master: e4943021ca9a48e2825025b3307e8d49 (127.3.252.67:44541)
I20260928 10:59:07.069794 13429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.67:44143
I20260928 10:59:07.071830 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:59:07.078020 13617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:07.076330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.3.252.126:40359)
0928 10:59:07.076500 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:07.076505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:07.076901 (+ 396us) server_negotiation.cc:407] Connection header received
0928 10:59:07.077151 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:07.077157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:07.077236 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:07.077325 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:07.077740 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33154: BlockingRecv error: recv got EOF from 127.0.0.1:33154 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:59:07.079181 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13488
I20260928 10:59:07.086519 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:07.080028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.3.252.65:44519)
0928 10:59:07.080511 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:07.080518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:07.080544 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:59:07.080605 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:07.080611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:07.080818 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:07.080947 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:07.081288 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.082327 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.083530 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.084073 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.084166 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:07.085095 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:07.085103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:07.085117 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:07.085719 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:07.085771 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:07.085797 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:07.085932 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:07.086030 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 10:59:07.086157 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260928 10:59:07.087942 13610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:07.167076 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13488
I20260928 10:59:07.180549 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13619
I20260928 10:59:07.187884 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:07.181402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.3.252.66:39841)
0928 10:59:07.181952 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:07.181959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:07.181986 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:59:07.182093 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:07.182101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:07.182320 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:07.182442 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:07.182776 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.183764 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.184944 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.185502 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.185598 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:07.186590 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:07.186598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:07.186612 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:07.187192 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:07.187238 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:07.187256 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:07.187338 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:07.187447 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 10:59:07.187557 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":270,"threads_started":1}
I20260928 10:59:07.189037 13741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:07.276767 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13619
I20260928 10:59:07.290105 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13749
I20260928 10:59:07.298413 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:07.291063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.3.252.67:44541)
0928 10:59:07.291681 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:07.291695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:07.291725 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:59:07.291784 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:07.291790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:07.292002 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:07.292143 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:07.292599 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.293730 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.295244 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:07.295861 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:07.295976 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:07.297015 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:07.297025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:07.297078 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:07.297660 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:07.297708 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:07.297728 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:07.297821 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:07.297985 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 10:59:07.298119 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
I20260928 10:59:07.299633 13871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:07.380327 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13749
I20260928 10:59:07.393325 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13398
I20260928 10:59:07.394425 13459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:07.493355 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13398
2026-09-28T10:59:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2331 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T10:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:59:07Z Disabled control of system clock
I20260928 10:59:07.555819 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:45359
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 10:59:07.866686 13895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:07.866995 13895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:07.867103 13895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:07.874573 13895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:59:07.874683 13895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:07.874749 13895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:07.874828 13895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:59:07.874886 13895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:59:07.884392 13895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:45359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:07.885670 13895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:07.887463 13895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:07.898576 13900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:07.898500 13903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:07.898526 13901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:07.899261 13895 server_base.cc:1034] running on GCE node
I20260928 10:59:07.899935 13895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:07.901262 13895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:07.902459 13895 hybrid_clock.cc:648] HybridClock initialized: now 1790593147902418 us; error 30 us; skew 500 ppm
I20260928 10:59:07.902791 13895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:07.904906 13895 webserver.cc:460] Webserver started at http://127.3.252.126:44233/ using document root <none> and password file <none>
I20260928 10:59:07.905552 13895 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:07.905640 13895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:07.905975 13895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:07.908710 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "33573e76fc4b42f7981bf8dda2193ac9"
format_stamp: "Formatted at 2026-09-28 10:59:07 on dist-test-slave-djjw"
server_key: "406bfa66f0645d7a9fa7583f40c799ec"
server_key_iv: "00bf3194dc11b4fbc226263b55facc74"
server_key_version: "encryptionkey@0"
I20260928 10:59:07.909451 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "33573e76fc4b42f7981bf8dda2193ac9"
format_stamp: "Formatted at 2026-09-28 10:59:07 on dist-test-slave-djjw"
server_key: "406bfa66f0645d7a9fa7583f40c799ec"
server_key_iv: "00bf3194dc11b4fbc226263b55facc74"
server_key_version: "encryptionkey@0"
I20260928 10:59:07.914368 13895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 10:59:07.917521 13909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:07.918819 13895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:59:07.918959 13895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "33573e76fc4b42f7981bf8dda2193ac9"
format_stamp: "Formatted at 2026-09-28 10:59:07 on dist-test-slave-djjw"
server_key: "406bfa66f0645d7a9fa7583f40c799ec"
server_key_iv: "00bf3194dc11b4fbc226263b55facc74"
server_key_version: "encryptionkey@0"
I20260928 10:59:07.919159 13895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:07.924650 13895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:07.925366 13895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:07.925617 13895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:07.941486 13895 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:45359
I20260928 10:59:07.941515 13960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:45359 every 8 connection(s)
I20260928 10:59:07.942732 13895 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:59:07.944540 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13895
I20260928 10:59:07.944744 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:59:07.945497 4081 external_mini_cluster.cc:1428] Setting key 6a41d04cda4e7750b58d72156aedb3c6
I20260928 10:59:07.947008 13961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:07.954250 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:07.947503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.3.252.126:45359)
0928 10:59:07.948491 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:07.948506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:07.948557 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:59:07.950324 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:07.950351 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:07.950879 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:07.951308 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:07.952272 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:07.952279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:07.952295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:07.952312 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:07.952396 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:07.952420 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:07.952635 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:07.952769 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 10:59:07.953034 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":301,"threads_started":1}
I20260928 10:59:07.959789 13961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9: Bootstrap starting.
I20260928 10:59:07.962685 13961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:07.963596 13961 log.cc:824] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:07.966300 13961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9: No bootstrap required, opened a new log
I20260928 10:59:07.973340 13961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33573e76fc4b42f7981bf8dda2193ac9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45359 } }
I20260928 10:59:07.973713 13961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:07.973788 13961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33573e76fc4b42f7981bf8dda2193ac9, State: Initialized, Role: FOLLOWER
I20260928 10:59:07.974340 13961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33573e76fc4b42f7981bf8dda2193ac9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45359 } }
I20260928 10:59:07.974567 13961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:59:07.974686 13961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:59:07.974833 13961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:07.977062 13961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33573e76fc4b42f7981bf8dda2193ac9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45359 } }
I20260928 10:59:07.977545 13961 leader_election.cc:302] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33573e76fc4b42f7981bf8dda2193ac9; no voters:
I20260928 10:59:07.978168 13961 leader_election.cc:288] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:59:07.978279 13966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:59:07.979017 13966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 33573e76fc4b42f7981bf8dda2193ac9, State: Running, Role: LEADER
I20260928 10:59:07.979547 13966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33573e76fc4b42f7981bf8dda2193ac9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45359 } }
I20260928 10:59:07.980154 13961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:59:07.982098 13968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33573e76fc4b42f7981bf8dda2193ac9. Latest consensus state: current_term: 1 leader_uuid: "33573e76fc4b42f7981bf8dda2193ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33573e76fc4b42f7981bf8dda2193ac9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45359 } } }
I20260928 10:59:07.982477 13968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:59:07.982995 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33573e76fc4b42f7981bf8dda2193ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33573e76fc4b42f7981bf8dda2193ac9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 45359 } } }
I20260928 10:59:07.983295 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:59:07.986868 13975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:59:07.991843 13975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:59:08.000388 13975 catalog_manager.cc:1347] Generated new cluster ID: 0d6769a71c5546f0b0ab822786522b9c
I20260928 10:59:08.000547 13975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:59:08.016512 13975 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:59:08.016891 13975 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:59:08.023844 13975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33573e76fc4b42f7981bf8dda2193ac9: Generated new TSK 0
I20260928 10:59:08.024628 13975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:59:08.039839 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:45359
--builtin_ntp_servers=127.3.252.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 10:59:08.375752 13985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:08.376026 13985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:08.376154 13985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:08.383374 13985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:08.383471 13985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:08.383616 13985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:59:08.392642 13985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:45359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:08.393939 13985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:08.395629 13985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:08.407650 13990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:08.407768 13991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:08.407735 13993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:08.408708 13985 server_base.cc:1034] running on GCE node
I20260928 10:59:08.409340 13985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:08.410326 13985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:08.411525 13985 hybrid_clock.cc:648] HybridClock initialized: now 1790593148411464 us; error 53 us; skew 500 ppm
I20260928 10:59:08.411824 13985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:08.414172 13985 webserver.cc:460] Webserver started at http://127.3.252.65:42453/ using document root <none> and password file <none>
I20260928 10:59:08.414855 13985 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:08.414953 13985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:08.415266 13985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:08.419364 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "7187a218c9eb4802b4c106019a4a8e2b"
format_stamp: "Formatted at 2026-09-28 10:59:08 on dist-test-slave-djjw"
server_key: "b6936510394f7be0b4ea9fe3f5808ae1"
server_key_iv: "bd08f5979cf4084c712d1d483de67dda"
server_key_version: "encryptionkey@0"
I20260928 10:59:08.420127 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "7187a218c9eb4802b4c106019a4a8e2b"
format_stamp: "Formatted at 2026-09-28 10:59:08 on dist-test-slave-djjw"
server_key: "b6936510394f7be0b4ea9fe3f5808ae1"
server_key_iv: "bd08f5979cf4084c712d1d483de67dda"
server_key_version: "encryptionkey@0"
I20260928 10:59:08.425290 13985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:59:08.428875 13999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:08.430290 13985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:59:08.430451 13985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "7187a218c9eb4802b4c106019a4a8e2b"
format_stamp: "Formatted at 2026-09-28 10:59:08 on dist-test-slave-djjw"
server_key: "b6936510394f7be0b4ea9fe3f5808ae1"
server_key_iv: "bd08f5979cf4084c712d1d483de67dda"
server_key_version: "encryptionkey@0"
I20260928 10:59:08.430675 13985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:08.435956 13985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:08.436710 13985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:08.436944 13985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:08.437793 13985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:08.439112 13985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:08.439204 13985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:08.439309 13985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:08.439378 13985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:08.468024 13985 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:44915
I20260928 10:59:08.468071 14111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:44915 every 8 connection(s)
I20260928 10:59:08.469241 13985 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:59:08.473982 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 13985
I20260928 10:59:08.474184 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:59:08.474947 4081 external_mini_cluster.cc:1428] Setting key 9cb94f3a136551ca9ec0b5c9dfaaa0cb
I20260928 10:59:08.477577 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:08.471574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:59607 (local address 127.3.252.126:45359)
0928 10:59:08.471963 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:08.471968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:08.473122 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 10:59:08.475302 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:08.475306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:08.475362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:08.475453 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:08.476884 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:08.476890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:08.476895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:08.476914 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:08.476965 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:08.476980 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:08.477085 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:08.477311 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 10:59:08.477362 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":201,"threads_started":1}
I20260928 10:59:08.478087 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:45359
--builtin_ntp_servers=127.3.252.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 10:59:08.478982 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:08.472138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:45359 (local address 127.3.252.65:59607)
0928 10:59:08.472981 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:08.473014 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:08.475009 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:08.475713 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:08.475723 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:08.475837 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:08.476553 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:59:08.476556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:08.476596 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:59:08.476693 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:08.477090 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:08.477100 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:08.477107 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:59:08.477386 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 10:59:08.477663 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":230,"threads_started":1}
I20260928 10:59:08.480547 14112 heartbeater.cc:347] Connected to a master server at 127.3.252.126:45359
I20260928 10:59:08.480845 14112 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:08.481469 14112 heartbeater.cc:511] Master 127.3.252.126:45359 requested a full tablet report, sending...
I20260928 10:59:08.483479 13926 ts_manager.cc:194] Registered new tserver with Master: 7187a218c9eb4802b4c106019a4a8e2b (127.3.252.65:44915)
W20260928 10:59:08.790330 14116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:08.790598 14116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:08.790693 14116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:08.797739 14116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:08.797865 14116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:08.798009 14116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:59:08.807024 14116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:45359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:08.808202 14116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:08.810397 14116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:08.822621 14124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:08.822690 14122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:08.822621 14121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:08.829406 14116 server_base.cc:1034] running on GCE node
I20260928 10:59:08.830075 14116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:08.831000 14116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:08.832196 14116 hybrid_clock.cc:648] HybridClock initialized: now 1790593148832156 us; error 37 us; skew 500 ppm
I20260928 10:59:08.832485 14116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:08.834745 14116 webserver.cc:460] Webserver started at http://127.3.252.66:43677/ using document root <none> and password file <none>
I20260928 10:59:08.835403 14116 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:08.835500 14116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:08.835806 14116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:08.838572 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "e2783c72ebcc45a5ba62bf35a75c75d2"
format_stamp: "Formatted at 2026-09-28 10:59:08 on dist-test-slave-djjw"
server_key: "ca7259d000b15876e4ba3122403ebfcc"
server_key_iv: "c4f329efaf974d11b02c35922ce431a3"
server_key_version: "encryptionkey@0"
I20260928 10:59:08.839308 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "e2783c72ebcc45a5ba62bf35a75c75d2"
format_stamp: "Formatted at 2026-09-28 10:59:08 on dist-test-slave-djjw"
server_key: "ca7259d000b15876e4ba3122403ebfcc"
server_key_iv: "c4f329efaf974d11b02c35922ce431a3"
server_key_version: "encryptionkey@0"
I20260928 10:59:08.844081 14116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:59:08.847240 14130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:08.848486 14116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 10:59:08.848625 14116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "e2783c72ebcc45a5ba62bf35a75c75d2"
format_stamp: "Formatted at 2026-09-28 10:59:08 on dist-test-slave-djjw"
server_key: "ca7259d000b15876e4ba3122403ebfcc"
server_key_iv: "c4f329efaf974d11b02c35922ce431a3"
server_key_version: "encryptionkey@0"
I20260928 10:59:08.848819 14116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:08.854138 14116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:08.854866 14116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:08.855094 14116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:08.855901 14116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:08.857184 14116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:08.857277 14116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:08.857379 14116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:08.857446 14116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:08.886854 14116 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:35157
I20260928 10:59:08.886920 14242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:35157 every 8 connection(s)
I20260928 10:59:08.888152 14116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:59:08.889297 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 14116
I20260928 10:59:08.889477 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:59:08.890036 4081 external_mini_cluster.cc:1428] Setting key e05873fa2a9b725cce901b086a1495e6
I20260928 10:59:08.892824 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:45359
--builtin_ntp_servers=127.3.252.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 10:59:08.896157 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:08.890780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:46669 (local address 127.3.252.126:45359)
0928 10:59:08.890971 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:08.890976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:08.892398 (+ 1422us) server_negotiation.cc:407] Connection header received
0928 10:59:08.894041 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:08.894047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:08.894119 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:08.894208 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:08.895480 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:08.895486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:08.895491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:08.895508 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:08.895536 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:08.895546 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:08.895642 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:08.895827 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 10:59:08.895888 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:59:08.897046 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:08.891324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:45359 (local address 127.3.252.66:46669)
0928 10:59:08.892260 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:08.892302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:08.893773 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:08.894431 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:08.894439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:08.894527 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:08.895205 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:59:08.895208 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:08.895243 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:59:08.895325 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:08.895626 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:08.895635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:08.895664 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 10:59:08.895896 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 10:59:08.896138 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":292,"threads_started":1}
I20260928 10:59:08.898320 14243 heartbeater.cc:347] Connected to a master server at 127.3.252.126:45359
I20260928 10:59:08.898612 14243 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:08.899298 14243 heartbeater.cc:511] Master 127.3.252.126:45359 requested a full tablet report, sending...
I20260928 10:59:08.900852 13926 ts_manager.cc:194] Registered new tserver with Master: e2783c72ebcc45a5ba62bf35a75c75d2 (127.3.252.66:35157)
W20260928 10:59:09.206979 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:09.207252 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:09.207374 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:09.214553 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:09.214668 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:09.214810 14246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:59:09.223877 14246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:45359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:09.225117 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:09.226802 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:09.238603 14251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:09.238673 14252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:09.238673 14254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:09.239624 14246 server_base.cc:1034] running on GCE node
I20260928 10:59:09.240175 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:09.241084 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:09.242298 14246 hybrid_clock.cc:648] HybridClock initialized: now 1790593149242249 us; error 34 us; skew 500 ppm
I20260928 10:59:09.242601 14246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:59:09.244727 14246 webserver.cc:460] Webserver started at http://127.3.252.67:46011/ using document root <none> and password file <none>
I20260928 10:59:09.245361 14246 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:09.245455 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:09.245761 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:09.248487 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "6797d07cbb164ed8bf0435c7f4021a0f"
format_stamp: "Formatted at 2026-09-28 10:59:09 on dist-test-slave-djjw"
server_key: "45fe13ed726b4a5b59e3806a50e30586"
server_key_iv: "33c3a1e597844e78389b1aee9e55c83a"
server_key_version: "encryptionkey@0"
I20260928 10:59:09.249251 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "6797d07cbb164ed8bf0435c7f4021a0f"
format_stamp: "Formatted at 2026-09-28 10:59:09 on dist-test-slave-djjw"
server_key: "45fe13ed726b4a5b59e3806a50e30586"
server_key_iv: "33c3a1e597844e78389b1aee9e55c83a"
server_key_version: "encryptionkey@0"
I20260928 10:59:09.254254 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:59:09.257483 14260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:09.258777 14246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:59:09.258924 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "6797d07cbb164ed8bf0435c7f4021a0f"
format_stamp: "Formatted at 2026-09-28 10:59:09 on dist-test-slave-djjw"
server_key: "45fe13ed726b4a5b59e3806a50e30586"
server_key_iv: "33c3a1e597844e78389b1aee9e55c83a"
server_key_version: "encryptionkey@0"
I20260928 10:59:09.259124 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:09.264375 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:09.265106 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:09.265484 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:09.266515 14246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:09.267828 14246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:09.267917 14246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:09.268019 14246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:09.268085 14246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:09.296741 14246 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:40567
I20260928 10:59:09.296804 14372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:40567 every 8 connection(s)
I20260928 10:59:09.298096 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:59:09.303932 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 14246
I20260928 10:59:09.304143 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:59:09.304585 4081 external_mini_cluster.cc:1428] Setting key 6fd439c75841607173c9aa407ac92fac
I20260928 10:59:09.306092 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:09.300599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:57155 (local address 127.3.252.126:45359)
0928 10:59:09.300773 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:09.300778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:09.302156 (+ 1378us) server_negotiation.cc:407] Connection header received
0928 10:59:09.303745 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:09.303752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:09.303823 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:09.303938 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:09.305381 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:09.305386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:09.305389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:09.305408 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:09.305441 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:09.305452 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:09.305565 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:09.305725 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 10:59:09.305785 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:59:09.307032 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:09.301033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:45359 (local address 127.3.252.67:57155)
0928 10:59:09.301973 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:09.302014 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:09.303470 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:09.304227 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:09.304239 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:09.304343 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:09.305002 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:59:09.305004 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:09.305042 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:59:09.305168 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:09.305543 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:09.305551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:09.305557 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:59:09.305813 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 10:59:09.306068 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":263,"threads_started":1}
I20260928 10:59:09.308205 14373 heartbeater.cc:347] Connected to a master server at 127.3.252.126:45359
I20260928 10:59:09.308459 14373 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:09.308981 14373 heartbeater.cc:511] Master 127.3.252.126:45359 requested a full tablet report, sending...
I20260928 10:59:09.310230 13926 ts_manager.cc:194] Registered new tserver with Master: 6797d07cbb164ed8bf0435c7f4021a0f (127.3.252.67:40567)
I20260928 10:59:09.319650 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:59:09.325378 14114 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:59:09.323928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.3.252.126:45359)
0928 10:59:09.324138 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:09.324145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:09.324566 (+ 421us) server_negotiation.cc:407] Connection header received
0928 10:59:09.324735 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:09.324739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:09.324790 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:09.324867 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:09.325134 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58416: BlockingRecv error: recv got EOF from 127.0.0.1:58416 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:59:09.325918 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13985
I20260928 10:59:09.330009 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:09.326986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.3.252.65:44915)
0928 10:59:09.327690 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:09.327699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:09.327738 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:59:09.327829 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:09.327837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:09.328133 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:09.328318 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:09.329255 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:09.329262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:09.329275 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:09.329288 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:09.329330 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:09.329347 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:09.329474 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:09.329619 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 10:59:09.329720 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":360,"threads_started":1}
I20260928 10:59:09.331228 14107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:09.410537 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13985
I20260928 10:59:09.423781 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 14116
I20260928 10:59:09.427246 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:09.424618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.3.252.66:35157)
0928 10:59:09.425127 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:09.425135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:09.425162 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:59:09.425220 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:09.425227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:09.425460 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:09.425612 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:09.426559 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:09.426565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:09.426576 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:09.426603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:09.426641 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:09.426655 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:09.426773 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:09.426896 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 10:59:09.426993 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260928 10:59:09.428434 14238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:09.508275 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 14116
I20260928 10:59:09.521555 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 14246
I20260928 10:59:09.525525 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:09.522551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.3.252.67:40567)
0928 10:59:09.523245 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:09.523252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:09.523282 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:59:09.523341 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:09.523348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:09.523609 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:09.523752 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:09.524820 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:09.524827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:09.524838 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:59:09.524854 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:09.524897 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:59:09.524910 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:09.525059 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:09.525141 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 10:59:09.525254 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":357,"threads_started":1}
I20260928 10:59:09.526769 14368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:09.606175 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 14246
I20260928 10:59:09.618708 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 13895
I20260928 10:59:09.619793 13956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:09.717059 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 13895
2026-09-28T10:59:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2223 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:59:09 dist-test-slave-djjw krb5kdc[14391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:59:09 dist-test-slave-djjw krb5kdc[14391](info): set up 2 sockets
Sep 28 10:59:09 dist-test-slave-djjw krb5kdc[14391](info): commencing operation
krb5kdc: starting...
W20260928 10:59:10.718820 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:59:10 dist-test-slave-djjw krb5kdc[14391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:59:10.931453 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40689
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:40689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:59:11.242244 14404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:11.242537 14404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:11.242654 14404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:11.249744 14404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:59:11.249889 14404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:11.249969 14404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:11.250043 14404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:59:11.250108 14404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:59:11.260066 14404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:40689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:11.261379 14404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:11.262617 14404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:11.273700 14409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:11.273661 14412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:11.273732 14410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:11.274279 14404 server_base.cc:1034] running on GCE node
I20260928 10:59:11.274963 14404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:11.276263 14404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:11.277493 14404 hybrid_clock.cc:648] HybridClock initialized: now 1790593151277446 us; error 31 us; skew 500 ppm
I20260928 10:59:11.277845 14404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:11 dist-test-slave-djjw krb5kdc[14391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593151, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:11.285542 14404 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:59:11.287081 14404 webserver.cc:460] Webserver started at http://127.3.252.126:44269/ using document root <none> and password file <none>
I20260928 10:59:11.287745 14404 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:11.287848 14404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:11.288093 14404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:11.291157 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "d454901a4a2342218f385d21ba235588"
format_stamp: "Formatted at 2026-09-28 10:59:11 on dist-test-slave-djjw"
server_key: "d5545fd827dc5e610461b48fa5328486"
server_key_iv: "50bc68a9c07678d90e02d7e99a8cff4b"
server_key_version: "encryptionkey@0"
I20260928 10:59:11.291872 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "d454901a4a2342218f385d21ba235588"
format_stamp: "Formatted at 2026-09-28 10:59:11 on dist-test-slave-djjw"
server_key: "d5545fd827dc5e610461b48fa5328486"
server_key_iv: "50bc68a9c07678d90e02d7e99a8cff4b"
server_key_version: "encryptionkey@0"
I20260928 10:59:11.296875 14404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:59:11.300130 14419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:11.301514 14404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 10:59:11.301697 14404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "d454901a4a2342218f385d21ba235588"
format_stamp: "Formatted at 2026-09-28 10:59:11 on dist-test-slave-djjw"
server_key: "d5545fd827dc5e610461b48fa5328486"
server_key_iv: "50bc68a9c07678d90e02d7e99a8cff4b"
server_key_version: "encryptionkey@0"
I20260928 10:59:11.301914 14404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:11.318825 14404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:11.323038 14404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:11.323308 14404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:11.339572 14404 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:40689
I20260928 10:59:11.339594 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:40689 every 8 connection(s)
I20260928 10:59:11.340909 14404 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:59:11.344481 14471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:11.351287 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 14404
I20260928 10:59:11.351512 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:59:11.352097 4081 external_mini_cluster.cc:1428] Setting key ff7e75f20df6744b2e4b9ea58f18aeac
I20260928 10:59:11.354663 14471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588: Bootstrap starting.
I20260928 10:59:11.357899 14471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:11.359161 14471 log.cc:824] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:11.362337 14471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588: No bootstrap required, opened a new log
Sep 28 10:59:11 dist-test-slave-djjw krb5kdc[14391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:11.371038 14471 raft_consensus.cc:348] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d454901a4a2342218f385d21ba235588" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40689 } }
I20260928 10:59:11.371490 14471 raft_consensus.cc:374] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:11.371590 14471 raft_consensus.cc:729] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d454901a4a2342218f385d21ba235588, State: Initialized, Role: FOLLOWER
I20260928 10:59:11.372244 14471 consensus_queue.cc:260] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d454901a4a2342218f385d21ba235588" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40689 } }
I20260928 10:59:11.372503 14471 raft_consensus.cc:388] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:59:11.372648 14471 raft_consensus.cc:482] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:59:11.372828 14471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:11.375144 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:11.353967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.3.252.126:40689)
0928 10:59:11.354690 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:11.354701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:11.354742 (+ 41us) server_negotiation.cc:407] Connection header received
0928 10:59:11.356566 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:11.356595 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:11.356970 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:11.357279 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:11.359096 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:11.360737 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:11.362557 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:11.363253 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:11.363393 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:11.368057 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:11.368068 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:11.368085 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:11.368132 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:11.370993 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:11.371763 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:11.371773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:11.371781 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:11.371958 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:11.372418 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:11.372421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:11.372423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:11.373039 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:11.373287 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:11.373683 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 10:59:11.374094 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":325,"threads_started":1}
I20260928 10:59:11.375540 14471 raft_consensus.cc:504] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d454901a4a2342218f385d21ba235588" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40689 } }
I20260928 10:59:11.376012 14471 leader_election.cc:302] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d454901a4a2342218f385d21ba235588; no voters:
I20260928 10:59:11.376591 14471 leader_election.cc:288] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:59:11.376718 14476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:59:11.378750 14471 sys_catalog.cc:564] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:59:11.380517 14476 raft_consensus.cc:686] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [term 1 LEADER]: Becoming Leader. State: Replica: d454901a4a2342218f385d21ba235588, State: Running, Role: LEADER
I20260928 10:59:11.381186 14476 consensus_queue.cc:237] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d454901a4a2342218f385d21ba235588" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40689 } }
I20260928 10:59:11.384416 14477 sys_catalog.cc:455] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d454901a4a2342218f385d21ba235588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d454901a4a2342218f385d21ba235588" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40689 } } }
I20260928 10:59:11.384727 14477 sys_catalog.cc:458] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [sys.catalog]: This master's current role is: LEADER
I20260928 10:59:11.383948 14478 sys_catalog.cc:455] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d454901a4a2342218f385d21ba235588. Latest consensus state: current_term: 1 leader_uuid: "d454901a4a2342218f385d21ba235588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d454901a4a2342218f385d21ba235588" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40689 } } }
I20260928 10:59:11.385259 14478 sys_catalog.cc:458] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588 [sys.catalog]: This master's current role is: LEADER
W20260928 10:59:11.389333 14490 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:59:11.389456 14490 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:59:11.389587 14491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:59:11.391129 14491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:59:11.398666 14491 catalog_manager.cc:1347] Generated new cluster ID: 773667dbce3a4dfe942f1b1019e5e4f0
I20260928 10:59:11.398789 14491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:59:11.415791 14491 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:59:11.417634 14491 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:59:11.436365 14491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588: Generated new TSK 0
I20260928 10:59:11.437438 14491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:59:11.501008 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40689
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:59:11.815701 14497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:11.815972 14497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:11.816061 14497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:11.823951 14497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:11.824059 14497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:11.824195 14497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:59:11.833861 14497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:11.835042 14497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:11.836229 14497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:11.848037 14505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:11.848057 14503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:11.848188 14502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:11.848860 14497 server_base.cc:1034] running on GCE node
I20260928 10:59:11.849395 14497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:11.850281 14497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:11.851461 14497 hybrid_clock.cc:648] HybridClock initialized: now 1790593151851409 us; error 42 us; skew 500 ppm
I20260928 10:59:11.851737 14497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:11 dist-test-slave-djjw krb5kdc[14391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593151, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:11.854696 14497 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:59:11.855978 14497 webserver.cc:460] Webserver started at http://127.3.252.65:39689/ using document root <none> and password file <none>
I20260928 10:59:11.856580 14497 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:11.856683 14497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:11.856918 14497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:11.859643 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "56c5fc68eadc43de80d580a5f797915c"
format_stamp: "Formatted at 2026-09-28 10:59:11 on dist-test-slave-djjw"
server_key: "233a2080bcef2bcd22d59ce00e6c94d5"
server_key_iv: "5f9948536d3d85c0e80a6bb819ab88aa"
server_key_version: "encryptionkey@0"
I20260928 10:59:11.860337 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "56c5fc68eadc43de80d580a5f797915c"
format_stamp: "Formatted at 2026-09-28 10:59:11 on dist-test-slave-djjw"
server_key: "233a2080bcef2bcd22d59ce00e6c94d5"
server_key_iv: "5f9948536d3d85c0e80a6bb819ab88aa"
server_key_version: "encryptionkey@0"
I20260928 10:59:11.865092 14497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 10:59:11.868371 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:11.869683 14497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:59:11.869850 14497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "56c5fc68eadc43de80d580a5f797915c"
format_stamp: "Formatted at 2026-09-28 10:59:11 on dist-test-slave-djjw"
server_key: "233a2080bcef2bcd22d59ce00e6c94d5"
server_key_iv: "5f9948536d3d85c0e80a6bb819ab88aa"
server_key_version: "encryptionkey@0"
I20260928 10:59:11.870072 14497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:11.890980 14497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:11.895097 14497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:11.895355 14497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:11.896230 14497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:11.897464 14497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:11.897536 14497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:11.897624 14497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:11.897665 14497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:11.924333 14497 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:36643
I20260928 10:59:11.924396 14624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:36643 every 8 connection(s)
I20260928 10:59:11.925530 14497 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:59:11.932821 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 14497
I20260928 10:59:11.933004 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:59:11.933652 4081 external_mini_cluster.cc:1428] Setting key 09100aaa96c501e708ffb6ca2446beff
Sep 28 10:59:11 dist-test-slave-djjw krb5kdc[14391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593151, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:11.944945 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:11.927918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:59353 (local address 127.3.252.126:40689)
0928 10:59:11.928337 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:11.928341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:11.929177 (+ 836us) server_negotiation.cc:407] Connection header received
0928 10:59:11.930942 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:11.930948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:11.931021 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:11.931111 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:11.932556 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:11.933345 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:11.935290 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:11.935778 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:11.935888 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:11.939671 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:11.939676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:11.939679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:11.939710 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:11.941968 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:11.942779 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:11.942785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:11.942789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:11.942931 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:11.943292 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:11.943296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:11.943299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:11.943624 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:11.943771 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:11.944390 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 10:59:11.944591 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":212,"threads_started":1}
I20260928 10:59:11.945739 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:11.928367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40689 (local address 127.3.252.65:59353)
0928 10:59:11.929029 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:11.929067 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:11.930689 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:11.931346 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:11.931354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:11.931848 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:11.932367 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:11.932385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:11.933486 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:11.933492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:11.935163 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:11.935177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:11.935987 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:11.935995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:11.936267 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:11.936925 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:11.936952 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:11.939448 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:11.942089 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:11.942098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:11.942115 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:11.942638 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:11.943025 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:11.943028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:11.943031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:11.943199 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:11.943721 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:11.943726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:11.944024 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 10:59:11.944257 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 10:59:11.944621 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260928 10:59:11.947162 14625 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40689
I20260928 10:59:11.947444 14625 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:11.948102 14625 heartbeater.cc:511] Master 127.3.252.126:40689 requested a full tablet report, sending...
I20260928 10:59:11.950086 14436 ts_manager.cc:194] Registered new tserver with Master: 56c5fc68eadc43de80d580a5f797915c (127.3.252.65:36643)
I20260928 10:59:11.951475 14436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:59353
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:59:11.981285 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40689
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:59:12.294598 14631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:12.294907 14631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:12.295038 14631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:12.302282 14631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:12.302407 14631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:12.302559 14631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:59:12.312384 14631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:12.313653 14631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:12.314922 14631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:12.327399 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:12.327316 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:12.327598 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:12.328218 14631 server_base.cc:1034] running on GCE node
I20260928 10:59:12.328756 14631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:12.329684 14631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:12.330894 14631 hybrid_clock.cc:648] HybridClock initialized: now 1790593152330831 us; error 56 us; skew 500 ppm
I20260928 10:59:12.331202 14631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:12 dist-test-slave-djjw krb5kdc[14391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:12.334280 14631 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:59:12.335741 14631 webserver.cc:460] Webserver started at http://127.3.252.66:42287/ using document root <none> and password file <none>
I20260928 10:59:12.336380 14631 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:12.336475 14631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:12.336716 14631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:12.340112 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "70d4d94b406d424498977b461360f8f5"
format_stamp: "Formatted at 2026-09-28 10:59:12 on dist-test-slave-djjw"
server_key: "509d20dcc163e4e2e71c898945a1d924"
server_key_iv: "bbd390cc3febe75d973554e7447c7f75"
server_key_version: "encryptionkey@0"
I20260928 10:59:12.341084 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "70d4d94b406d424498977b461360f8f5"
format_stamp: "Formatted at 2026-09-28 10:59:12 on dist-test-slave-djjw"
server_key: "509d20dcc163e4e2e71c898945a1d924"
server_key_iv: "bbd390cc3febe75d973554e7447c7f75"
server_key_version: "encryptionkey@0"
I20260928 10:59:12.347453 14631 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 10:59:12.351627 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:12.353286 14631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:59:12.353436 14631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "70d4d94b406d424498977b461360f8f5"
format_stamp: "Formatted at 2026-09-28 10:59:12 on dist-test-slave-djjw"
server_key: "509d20dcc163e4e2e71c898945a1d924"
server_key_iv: "bbd390cc3febe75d973554e7447c7f75"
server_key_version: "encryptionkey@0"
I20260928 10:59:12.353643 14631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:12.384215 14631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:12.388167 14631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:12.388415 14631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:12.389317 14631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:12.390642 14631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:12.390713 14631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:12.390805 14631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:12.390861 14631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:12.420455 14631 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:39673
I20260928 10:59:12.420500 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:39673 every 8 connection(s)
I20260928 10:59:12.421696 14631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:59:12.431720 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 14631
I20260928 10:59:12.431906 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:59:12.432336 4081 external_mini_cluster.cc:1428] Setting key 7ab70af6eb49cec8cd36a3a36f8bf30e
Sep 28 10:59:12 dist-test-slave-djjw krb5kdc[14391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:12.442642 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.424200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:42813 (local address 127.3.252.126:40689)
0928 10:59:12.424416 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:12.424422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:12.425573 (+ 1151us) server_negotiation.cc:407] Connection header received
0928 10:59:12.427765 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:12.427771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:12.427852 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:12.427966 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:12.429980 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.430735 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.432328 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.432911 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.433019 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.436830 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:12.436837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:12.436841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:12.436878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:12.439700 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:12.440422 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:12.440427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:12.440430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:12.440561 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:12.440928 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:12.440932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:12.440935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:12.441260 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:12.441407 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:12.442099 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 10:59:12.442320 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:59:12.443617 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.424696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40689 (local address 127.3.252.66:42813)
0928 10:59:12.425415 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:12.425452 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:12.427082 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:12.428401 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:12.428414 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:12.429112 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:12.429751 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.429774 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.430859 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.430863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.432159 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.432170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.433207 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.433215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.433430 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.434164 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:12.434188 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:12.436566 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:12.439840 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:12.439848 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:12.439863 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:12.440300 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:12.440648 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:12.440651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:12.440653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:12.440830 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:12.441369 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:12.441375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:12.441700 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 10:59:12.441967 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 10:59:12.442369 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":263,"threads_started":1}
I20260928 10:59:12.444947 14759 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40689
I20260928 10:59:12.445231 14759 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:12.445968 14759 heartbeater.cc:511] Master 127.3.252.126:40689 requested a full tablet report, sending...
I20260928 10:59:12.447391 14436 ts_manager.cc:194] Registered new tserver with Master: 70d4d94b406d424498977b461360f8f5 (127.3.252.66:39673)
I20260928 10:59:12.448159 14436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:42813
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:59:12.481415 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40689
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:59:12.788443 14764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:12.788765 14764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:12.788911 14764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:12.796072 14764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:12.796203 14764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:12.796380 14764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:59:12.806012 14764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:12.807276 14764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:12.808508 14764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:12.820629 14769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:12.820663 14772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:12.820705 14770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:12.821471 14764 server_base.cc:1034] running on GCE node
I20260928 10:59:12.822203 14764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:12.823307 14764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:12.824506 14764 hybrid_clock.cc:648] HybridClock initialized: now 1790593152824448 us; error 43 us; skew 500 ppm
I20260928 10:59:12.824864 14764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:12 dist-test-slave-djjw krb5kdc[14391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:12.827752 14764 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:59:12.829128 14764 webserver.cc:460] Webserver started at http://127.3.252.67:43617/ using document root <none> and password file <none>
I20260928 10:59:12.829813 14764 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:12.829954 14764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:12.830206 14764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:59:12.832958 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "1cf5a5ee91d142fbbded2c20bbe52bde"
format_stamp: "Formatted at 2026-09-28 10:59:12 on dist-test-slave-djjw"
server_key: "42bc9e8beff9977b2af087d0717d439f"
server_key_iv: "3846ac73cea1425a6e6433150458782e"
server_key_version: "encryptionkey@0"
I20260928 10:59:12.833635 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "1cf5a5ee91d142fbbded2c20bbe52bde"
format_stamp: "Formatted at 2026-09-28 10:59:12 on dist-test-slave-djjw"
server_key: "42bc9e8beff9977b2af087d0717d439f"
server_key_iv: "3846ac73cea1425a6e6433150458782e"
server_key_version: "encryptionkey@0"
I20260928 10:59:12.838389 14764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:59:12.841552 14779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:12.842864 14764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:59:12.842994 14764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "1cf5a5ee91d142fbbded2c20bbe52bde"
format_stamp: "Formatted at 2026-09-28 10:59:12 on dist-test-slave-djjw"
server_key: "42bc9e8beff9977b2af087d0717d439f"
server_key_iv: "3846ac73cea1425a6e6433150458782e"
server_key_version: "encryptionkey@0"
I20260928 10:59:12.843153 14764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:12.866506 14764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:12.870178 14764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:12.870416 14764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:12.871384 14764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:12.872586 14764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:12.872655 14764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:12.872738 14764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:12.872802 14764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:12.902004 14764 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:42587
I20260928 10:59:12.902048 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:42587 every 8 connection(s)
I20260928 10:59:12.903187 14764 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:59:12.911500 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 14764
I20260928 10:59:12.911700 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:59:12.912225 4081 external_mini_cluster.cc:1428] Setting key 6896b4a1c5d3bd5100daadfa5b5769b5
Sep 28 10:59:12 dist-test-slave-djjw krb5kdc[14391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:12.921649 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.905511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:35827 (local address 127.3.252.126:40689)
0928 10:59:12.905710 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:12.905715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:12.906799 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 10:59:12.908517 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:12.908521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:12.908584 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:12.908666 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:12.910051 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.910797 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.912883 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.913351 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.913470 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.916636 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:12.916641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:12.916644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:12.916683 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:12.918852 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:12.919518 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:12.919522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:12.919524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:12.919640 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:12.920068 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:12.920071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:12.920073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:12.920336 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:12.920457 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:12.921112 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 10:59:12.921326 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 10:59:12.922600 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.905969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40689 (local address 127.3.252.67:35827)
0928 10:59:12.906641 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:12.906687 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:12.908291 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:12.908844 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:12.908852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:12.909348 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:12.909918 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.909936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.910913 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.910917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.912736 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.912748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.913559 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.913566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.913793 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.914438 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:12.914459 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:12.916447 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:12.918975 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:12.918985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:12.919001 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:12.919405 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:12.919745 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:12.919750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:12.919754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:12.919953 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:12.920451 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:12.920459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:12.920777 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 10:59:12.921009 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 10:59:12.921504 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":255,"threads_started":1}
I20260928 10:59:12.923871 14892 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40689
I20260928 10:59:12.924245 14892 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:12.925104 14892 heartbeater.cc:511] Master 127.3.252.126:40689 requested a full tablet report, sending...
I20260928 10:59:12.926757 14436 ts_manager.cc:194] Registered new tserver with Master: 1cf5a5ee91d142fbbded2c20bbe52bde (127.3.252.67:42587)
I20260928 10:59:12.927870 14436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:35827
I20260928 10:59:12.927971 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:59:12.944585 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.932801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.3.252.126:40689)
0928 10:59:12.932953 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:12.932960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:12.933383 (+ 423us) server_negotiation.cc:407] Connection header received
0928 10:59:12.933549 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:12.933555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:12.933619 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:12.933696 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:12.935292 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.936014 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.937222 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.937607 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.937712 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.939858 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:12.939864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:12.939867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:12.939898 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:12.942168 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:12.942752 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:12.942757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:12.942761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:12.942895 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:12.943234 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:12.943239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:12.943242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:12.943538 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:12.943758 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:12.944061 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 10:59:12.944285 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:59:12.948561 14436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 10:59:12.954355 14625 heartbeater.cc:503] Master 127.3.252.126:40689 was elected leader, sending a full tablet report...
W20260928 10:59:12.955329 14436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:59:12.991091 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.967764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.3.252.66:39673)
0928 10:59:12.983878 (+ 16114us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:12.983886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:12.983912 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:59:12.984001 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:12.984007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:12.984227 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:12.984396 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:12.985186 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.986689 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.988982 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.990005 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.990137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.990339 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:12.990610 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 10:59:12.990786 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15965,"thread_start_us":15858,"threads_started":1}
I20260928 10:59:12.992165 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.967278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:39673 (local address 127.0.0.1:33066)
0928 10:59:12.981954 (+ 14676us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:12.981975 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:12.982115 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:12.984450 (+ 2335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:12.984457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:12.984488 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:12.984987 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.984999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.986847 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.986852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.988773 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.988789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.990184 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.990190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.990373 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.990421 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 10:59:12.990563 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 10:59:12.991718 (+ 1155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14491,"mutex_wait_us":5,"thread_start_us":626,"threads_started":1}
I20260928 10:59:12.992435 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.967121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:42587 (local address 127.0.0.1:42012)
0928 10:59:12.967947 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:12.967988 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:12.968180 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:12.984876 (+ 16696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:12.984881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:12.984907 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:12.985449 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.985462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.987813 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.987818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.989949 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.989959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.991724 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.991731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.991852 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.991875 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 10:59:12.991926 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 10:59:12.992016 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":25,"thread_start_us":15078,"threads_started":1}
I20260928 10:59:12.992975 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.967229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.3.252.67:42587)
0928 10:59:12.984327 (+ 17098us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:12.984334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:12.984366 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:59:12.984449 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:12.984455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:12.984665 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:12.985640 (+ 975us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:12.985919 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.987679 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.990079 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.991504 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.992242 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.992463 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:12.992532 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 10:59:12.992689 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16940,"thread_start_us":16779,"threads_started":1}
I20260928 10:59:12.993559 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.969063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.3.252.65:36643)
0928 10:59:12.982894 (+ 13831us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:12.982903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:12.983422 (+ 519us) server_negotiation.cc:407] Connection header received
0928 10:59:12.983622 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:12.983631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:12.984057 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:12.984422 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:12.985659 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.987224 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.989803 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.991104 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.991231 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.991443 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:12.992296 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 10:59:12.993257 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13650,"thread_start_us":13501,"threads_started":1}
I20260928 10:59:12.993721 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:12.968721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:36643 (local address 127.0.0.1:35374)
0928 10:59:12.983289 (+ 14568us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:12.983307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:12.983429 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:12.984668 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:12.984673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:12.984698 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:12.985391 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.985407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.987380 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.987385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.989638 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:12.989653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:12.991633 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:12.991639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:12.991820 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:12.991856 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:59:12.991931 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 10:59:12.992052 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14462,"mutex_wait_us":74,"thread_start_us":14351,"threads_started":1}
I20260928 10:59:12.996534 14560 tablet_service.cc:1467] Processing CreateTablet for tablet 6be92208c83b4858aa01d6e94e773661 (DEFAULT_TABLE table=test-table [id=4a09ae8bd2f247069026117401ad9296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:59:12.996506 14827 tablet_service.cc:1467] Processing CreateTablet for tablet 6be92208c83b4858aa01d6e94e773661 (DEFAULT_TABLE table=test-table [id=4a09ae8bd2f247069026117401ad9296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:59:12.996506 14694 tablet_service.cc:1467] Processing CreateTablet for tablet 6be92208c83b4858aa01d6e94e773661 (DEFAULT_TABLE table=test-table [id=4a09ae8bd2f247069026117401ad9296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:59:12.997946 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be92208c83b4858aa01d6e94e773661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:12.997963 14827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be92208c83b4858aa01d6e94e773661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:12.997963 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be92208c83b4858aa01d6e94e773661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:13.009020 14910 tablet_bootstrap.cc:492] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Bootstrap starting.
I20260928 10:59:13.012192 14910 tablet_bootstrap.cc:654] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:13.013037 14912 tablet_bootstrap.cc:492] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Bootstrap starting.
I20260928 10:59:13.013437 14910 log.cc:824] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:13.014892 14911 tablet_bootstrap.cc:492] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Bootstrap starting.
I20260928 10:59:13.016571 14910 tablet_bootstrap.cc:492] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: No bootstrap required, opened a new log
I20260928 10:59:13.016737 14912 tablet_bootstrap.cc:654] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:13.016829 14910 ts_tablet_manager.cc:1397] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 10:59:13.017922 14912 log.cc:824] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:13.018291 14911 tablet_bootstrap.cc:654] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:13.019411 14911 log.cc:824] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:13.020860 14912 tablet_bootstrap.cc:492] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: No bootstrap required, opened a new log
I20260928 10:59:13.021090 14912 ts_tablet_manager.cc:1397] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 10:59:13.022229 14911 tablet_bootstrap.cc:492] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: No bootstrap required, opened a new log
I20260928 10:59:13.022469 14911 ts_tablet_manager.cc:1397] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 10:59:13.026990 14910 raft_consensus.cc:348] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.027453 14910 raft_consensus.cc:374] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:13.027583 14910 raft_consensus.cc:729] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c5fc68eadc43de80d580a5f797915c, State: Initialized, Role: FOLLOWER
I20260928 10:59:13.027884 14912 raft_consensus.cc:348] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.028240 14912 raft_consensus.cc:374] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:13.028332 14912 raft_consensus.cc:729] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d4d94b406d424498977b461360f8f5, State: Initialized, Role: FOLLOWER
I20260928 10:59:13.028216 14910 consensus_queue.cc:260] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.028893 14912 consensus_queue.cc:260] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.030095 14910 ts_tablet_manager.cc:1428] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260928 10:59:13.031029 14759 heartbeater.cc:503] Master 127.3.252.126:40689 was elected leader, sending a full tablet report...
I20260928 10:59:13.031798 14912 ts_tablet_manager.cc:1428] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260928 10:59:13.032296 14911 raft_consensus.cc:348] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.032799 14911 raft_consensus.cc:374] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:13.032948 14911 raft_consensus.cc:729] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf5a5ee91d142fbbded2c20bbe52bde, State: Initialized, Role: FOLLOWER
I20260928 10:59:13.033617 14911 consensus_queue.cc:260] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.034735 14892 heartbeater.cc:503] Master 127.3.252.126:40689 was elected leader, sending a full tablet report...
I20260928 10:59:13.035377 14911 ts_tablet_manager.cc:1428] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 10:59:13.036509 14916 raft_consensus.cc:482] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:59:13.036759 14916 raft_consensus.cc:504] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.038604 14916 leader_election.cc:288] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cf5a5ee91d142fbbded2c20bbe52bde (127.3.252.67:42587), 70d4d94b406d424498977b461360f8f5 (127.3.252.66:39673)
I20260928 10:59:13.045635 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.038838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:39673 (local address 127.3.252.65:34763)
0928 10:59:13.039490 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:13.039509 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:13.039678 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:13.040032 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:13.040035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:13.040055 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:13.040364 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:13.040371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.041716 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.041719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:13.043608 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:13.043623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.045039 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.045044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:13.045174 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.045200 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:59:13.045282 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 10:59:13.045369 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":346,"threads_started":1}
I20260928 10:59:13.045715 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.039323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:34763 (local address 127.3.252.66:39673)
0928 10:59:13.039492 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:13.039498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:13.039579 (+ 81us) server_negotiation.cc:407] Connection header received
0928 10:59:13.039823 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:13.039829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:13.039909 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:13.039998 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:13.040476 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.041597 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.043758 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.044825 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.044970 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.045041 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:13.045324 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 10:59:13.045451 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:59:13.046703 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be92208c83b4858aa01d6e94e773661" candidate_uuid: "56c5fc68eadc43de80d580a5f797915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d4d94b406d424498977b461360f8f5" is_pre_election: true
I20260928 10:59:13.047302 14714 raft_consensus.cc:2454] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c5fc68eadc43de80d580a5f797915c in term 0.
I20260928 10:59:13.047997 14513 leader_election.cc:302] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c5fc68eadc43de80d580a5f797915c, 70d4d94b406d424498977b461360f8f5; no voters:
I20260928 10:59:13.048272 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.039432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:55585 (local address 127.3.252.67:42587)
0928 10:59:13.039639 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:13.039644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:13.042486 (+ 2842us) server_negotiation.cc:407] Connection header received
0928 10:59:13.042602 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:13.042607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:13.042676 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:13.042790 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:13.043239 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.044110 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.046340 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.047494 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.047653 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.047724 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:13.048005 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 10:59:13.048100 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:59:13.048482 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.038909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:42587 (local address 127.3.252.65:55585)
0928 10:59:13.042386 (+ 3477us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:13.042403 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:13.042500 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:13.042798 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:13.042801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:13.042819 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:13.043106 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:13.043113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.044238 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.044243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:13.046175 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:13.046188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.047696 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.047702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:13.047859 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.047890 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:59:13.047987 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 10:59:13.048114 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3362,"thread_start_us":3135,"threads_started":1}
I20260928 10:59:13.048455 14916 raft_consensus.cc:2781] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:59:13.048588 14916 raft_consensus.cc:482] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:59:13.048688 14916 raft_consensus.cc:3037] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:13.049405 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be92208c83b4858aa01d6e94e773661" candidate_uuid: "56c5fc68eadc43de80d580a5f797915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" is_pre_election: true
I20260928 10:59:13.049932 14847 raft_consensus.cc:2454] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c5fc68eadc43de80d580a5f797915c in term 0.
I20260928 10:59:13.051303 14916 raft_consensus.cc:504] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.051868 14916 leader_election.cc:288] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [CANDIDATE]: Term 1 election: Requested vote from peers 1cf5a5ee91d142fbbded2c20bbe52bde (127.3.252.67:42587), 70d4d94b406d424498977b461360f8f5 (127.3.252.66:39673)
I20260928 10:59:13.052265 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be92208c83b4858aa01d6e94e773661" candidate_uuid: "56c5fc68eadc43de80d580a5f797915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde"
I20260928 10:59:13.052332 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be92208c83b4858aa01d6e94e773661" candidate_uuid: "56c5fc68eadc43de80d580a5f797915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d4d94b406d424498977b461360f8f5"
I20260928 10:59:13.052461 14847 raft_consensus.cc:3037] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:13.052512 14714 raft_consensus.cc:3037] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:13.055567 14714 raft_consensus.cc:2454] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c5fc68eadc43de80d580a5f797915c in term 1.
I20260928 10:59:13.055558 14847 raft_consensus.cc:2454] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c5fc68eadc43de80d580a5f797915c in term 1.
I20260928 10:59:13.055933 14513 leader_election.cc:302] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c5fc68eadc43de80d580a5f797915c, 70d4d94b406d424498977b461360f8f5; no voters:
I20260928 10:59:13.056191 14916 raft_consensus.cc:2781] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:59:13.056593 14916 raft_consensus.cc:686] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 1 LEADER]: Becoming Leader. State: Replica: 56c5fc68eadc43de80d580a5f797915c, State: Running, Role: LEADER
I20260928 10:59:13.057020 14916 consensus_queue.cc:237] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } }
I20260928 10:59:13.061695 14434 catalog_manager.cc:5495] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c reported cstate change: term changed from 0 to 1, leader changed from <none> to 56c5fc68eadc43de80d580a5f797915c (127.3.252.65). New cstate: current_term: 1 leader_uuid: "56c5fc68eadc43de80d580a5f797915c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c5fc68eadc43de80d580a5f797915c" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36643 } health_report { overall_health: HEALTHY } } }
I20260928 10:59:13.080013 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.075262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.3.252.65:36643)
0928 10:59:13.075456 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:13.075462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:13.075510 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:59:13.075672 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:13.075675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:13.075762 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:13.075877 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:59:13.076157 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.076837 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.078089 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.078519 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.078606 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.079013 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:59:13.079516 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:59:13.079616 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:13.079714 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 10:59:13.079778 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 10:59:13.087373 14847 raft_consensus.cc:1261] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 1 FOLLOWER]: Refusing update from remote peer 56c5fc68eadc43de80d580a5f797915c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:59:13.087368 14714 raft_consensus.cc:1261] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 1 FOLLOWER]: Refusing update from remote peer 56c5fc68eadc43de80d580a5f797915c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:59:13.088243 14921 consensus_queue.cc:1035] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d4d94b406d424498977b461360f8f5" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:59:13.088572 14916 consensus_queue.cc:1035] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf5a5ee91d142fbbded2c20bbe52bde" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:59:13.101642 14924 mvcc.cc:205] Tried to move back new op lower bound from 7334269555036868608 to 7334269554925215744. Current Snapshot: MvccSnapshot[applied={T|T < 7334269555036868608 or (T in {7334269555036868608})}]
I20260928 10:59:13.107931 14925 mvcc.cc:205] Tried to move back new op lower bound from 7334269555036868608 to 7334269554925215744. Current Snapshot: MvccSnapshot[applied={T|T < 7334269555036868608}]
I20260928 10:59:13.112368 14434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:59:13.112644 14434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:59:13.116523 14434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d454901a4a2342218f385d21ba235588: Sending DeleteTablet for 3 replicas of tablet 6be92208c83b4858aa01d6e94e773661
I20260928 10:59:13.117573 14827 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be92208c83b4858aa01d6e94e773661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:59:13 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:42012
I20260928 10:59:13.117688 14560 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be92208c83b4858aa01d6e94e773661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:59:13 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:35374
I20260928 10:59:13.117941 14694 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be92208c83b4858aa01d6e94e773661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:59:13 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:33066
I20260928 10:59:13.120028 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 14497
I20260928 10:59:13.128999 14935 tablet_replica.cc:330] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: stopping tablet replica
I20260928 10:59:13.129453 14935 raft_consensus.cc:2229] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:59:13.129747 14934 tablet_replica.cc:330] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: stopping tablet replica
I20260928 10:59:13.130110 14935 raft_consensus.cc:2258] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:59:13.130151 14934 raft_consensus.cc:2229] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:59:13.130717 14934 raft_consensus.cc:2258] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:59:13.130877 14936 tablet_replica.cc:330] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: stopping tablet replica
I20260928 10:59:13.131220 14936 raft_consensus.cc:2229] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:59:13.131559 14936 raft_consensus.cc:2258] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:59:13.133755 14936 ts_tablet_manager.cc:1905] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:59:13.133744 14934 ts_tablet_manager.cc:1905] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:59:13.135133 14935 ts_tablet_manager.cc:1905] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 10:59:13 dist-test-slave-djjw krb5kdc[14391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:59:13.140322 14934 ts_tablet_manager.cc:1918] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:59:13.140506 14934 log.cc:1196] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/6be92208c83b4858aa01d6e94e773661
I20260928 10:59:13.141007 14934 ts_tablet_manager.cc:1939] T 6be92208c83b4858aa01d6e94e773661 P 1cf5a5ee91d142fbbded2c20bbe52bde: Deleting consensus metadata
I20260928 10:59:13.142899 14421 catalog_manager.cc:4841] TS 1cf5a5ee91d142fbbded2c20bbe52bde (127.3.252.67:42587): tablet 6be92208c83b4858aa01d6e94e773661 (table test-table [id=4a09ae8bd2f247069026117401ad9296]) successfully deleted
I20260928 10:59:13.145324 14936 ts_tablet_manager.cc:1918] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:59:13.145466 14936 log.cc:1196] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/6be92208c83b4858aa01d6e94e773661
I20260928 10:59:13.146598 14936 ts_tablet_manager.cc:1939] T 6be92208c83b4858aa01d6e94e773661 P 70d4d94b406d424498977b461360f8f5: Deleting consensus metadata
I20260928 10:59:13.147739 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.121358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.3.252.65:36643)
0928 10:59:13.125710 (+ 4352us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:13.125717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:13.126322 (+ 605us) server_negotiation.cc:407] Connection header received
0928 10:59:13.126588 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:13.126594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:13.126677 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:13.126770 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:13.130738 (+ 3968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.131865 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.133892 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.134595 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.134697 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.139882 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:13.139899 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:13.139918 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:13.139975 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:13.144070 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:13.145025 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:13.145035 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:13.145038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:13.145219 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:13.145691 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:13.145696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:13.145703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:13.146219 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:13.146534 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:13.147096 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 10:59:13.147349 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4201}
I20260928 10:59:13.148573 14935 ts_tablet_manager.cc:1918] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:59:13.148602 14420 catalog_manager.cc:4841] TS 70d4d94b406d424498977b461360f8f5 (127.3.252.66:39673): tablet 6be92208c83b4858aa01d6e94e773661 (table test-table [id=4a09ae8bd2f247069026117401ad9296]) successfully deleted
I20260928 10:59:13.148765 14935 log.cc:1196] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/6be92208c83b4858aa01d6e94e773661
I20260928 10:59:13.149178 14620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:13.149259 14935 ts_tablet_manager.cc:1939] T 6be92208c83b4858aa01d6e94e773661 P 56c5fc68eadc43de80d580a5f797915c: Deleting consensus metadata
I20260928 10:59:13.245443 14420 catalog_manager.cc:4841] TS 56c5fc68eadc43de80d580a5f797915c (127.3.252.65:36643): tablet 6be92208c83b4858aa01d6e94e773661 (table test-table [id=4a09ae8bd2f247069026117401ad9296]) successfully deleted
I20260928 10:59:13.245519 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 14497
I20260928 10:59:13.261281 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 14631
Sep 28 10:59:13 dist-test-slave-djjw krb5kdc[14391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:59:13.274950 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.261938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.3.252.66:39673)
0928 10:59:13.262096 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:13.262102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:13.262164 (+ 62us) server_negotiation.cc:407] Connection header received
0928 10:59:13.262358 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:13.262363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:13.262434 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:13.262513 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:13.263683 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.264406 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.265588 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.266051 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.266153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.269948 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:13.269961 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:13.269977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:13.270011 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:13.272429 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:13.273051 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:13.273057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:13.273060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:13.273170 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:13.273499 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:13.273503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:13.273505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:13.273889 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:13.274147 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:13.274434 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 10:59:13.274637 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:59:13.275538 14754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:13.362430 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 14631
I20260928 10:59:13.377319 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 14764
Sep 28 10:59:13 dist-test-slave-djjw krb5kdc[14391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 10:59:13.391462 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:13.378056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.3.252.67:42587)
0928 10:59:13.378215 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:13.378220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:13.378238 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:59:13.378376 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:13.378380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:13.378450 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:13.378531 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:13.379725 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.380498 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.381811 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:13.382210 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:13.382307 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:13.386255 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:13.386270 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:13.386287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:13.386323 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:13.388935 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:13.389527 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:13.389535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:13.389539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:13.389684 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:13.390061 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:13.390067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:13.390070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:13.390505 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:13.390730 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:13.390954 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 10:59:13.391114 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 10:59:13.392046 14887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:13.478564 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 14764
I20260928 10:59:13.492063 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 14404
I20260928 10:59:13.493247 14466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:13.598380 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 14404
2026-09-28T10:59:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3882 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:59:13 dist-test-slave-djjw krb5kdc[14944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:59:13 dist-test-slave-djjw krb5kdc[14944](info): set up 2 sockets
Sep 28 10:59:13 dist-test-slave-djjw krb5kdc[14944](info): commencing operation
krb5kdc: starting...
W20260928 10:59:14.610740 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 10:59:14 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:59:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/rangeradmin_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/rangeradmin_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/rangerlookup_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/rangerlookup_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
I20260928 10:59:14.924741 4081 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktiZmwa/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/postgres -l logfile start
2026-09-28 10:59:17.343 UTC [14974] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 10:59:17.343 UTC [14974] LOG: listening on IPv4 address "127.3.252.84", port 57173
2026-09-28 10:59:17.347 UTC [14974] LOG: listening on Unix socket "/tmp/.s.PGSQL.57173"
2026-09-28 10:59:17.368 UTC [14977] LOG: database system was shut down at 2026-09-28 10:59:15 UTC
2026-09-28 10:59:17.372 UTC [14974] LOG: database system is ready to accept connections
I20260928 10:59:18.258288 4081 mini_postgres.cc:96] Postgres bound to 57173
2026-09-28 10:59:18.274 UTC [14985] FATAL: database "slave" does not exist
127.3.252.84:57173 - accepting connections
I20260928 10:59:18.274938 4081 mini_ranger.cc:161] Starting Ranger...
I20260928 10:59:18.299780 4081 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 10:59:18.767308 4081 mini_ranger.cc:87] Created ranger Postgres database
I20260928 10:59:18.767414 4081 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-admin
2026-09-28 10:59:19,045 [I] DB FLAVOR :POSTGRES
2026-09-28 10:59:19,046 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:59:19,046 [I] Checking connection..
2026-09-28 10:59:19,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 10:59:19,294 [I] Checking connection passed.
2026-09-28 10:59:19,295 [I] --------- Verifying version history table ---------
2026-09-28 10:59:19,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:59:19,541 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:59:19,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:59:19,791 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:59:19,791 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 10:59:19,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 10:59:20.027 UTC [15049] WARNING: there is no transaction in progress
2026-09-28 10:59:20,033 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 10:59:20,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:59:20,269 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 10:59:20,269 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:59:20,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 10:59:20,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 10:59:20,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-djjw.c.gcp-upstream.internal','N') ;"
2026-09-28 10:59:20,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:59:21,241 [I] Table x_portal_user does not exist in database ranger
2026-09-28 10:59:21,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:59:21,466 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 10:59:21,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:59:21,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:59:21,945 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 10:59:21,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 10:59:22.886 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:22.901 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:22.917 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:22.932 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:22.951 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:22.970 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.026 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.033 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.047 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.057 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.068 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.080 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.090 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.096 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.103 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23.110 UTC [15157] WARNING: there is no transaction in progress
2026-09-28 10:59:23,736 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 10:59:23,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 10:59:23,984 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 10:59:23,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:59:24,224 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 10:59:24,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:59:24,454 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 10:59:24,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:59:24,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:59:24,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-djjw.c.gcp-upstream.internal';"
2026-09-28 10:59:25,205 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 10:59:25,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 10:59:25,430 [I] DB_PATCHES have already been applied
I20260928 10:59:25.436385 4081 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktiZmwa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 10:59:25.436477 4081 mini_ranger.cc:193] Using host: 127.3.252.84
I20260928 10:59:25.450551 4081 mini_ranger.cc:227] Ranger admin URL: http://127.3.252.84:60707
Sep 28, 2026 10:59:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:59:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:59:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:59:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:59:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:59:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:59:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 10:59:26.476629 4081 mini_ranger.cc:160] Time spent starting Ranger: real 8.202s user 0.000s sys 0.038s
Sep 28, 2026 10:59:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:59:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 10:59:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:59:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:59:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 10:59:47.731343 4081 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/rangerkms_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/rangerkms_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.252.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.252.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 10:59:47 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593187, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 10:59:47.886739 4081 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 10:59:47.906219 4081 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 10:59:48.822207 4081 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 10:59:48.822335 4081 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms
I20260928 10:59:48.822458 4081 mini_ranger_kms.cc:227] Using postgres at 127.3.252.84:57173
2026-09-28 10:59:49,216 [I] DB FLAVOR :POSTGRES
2026-09-28 10:59:49,216 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:59:49,216 [I] Checking connection
2026-09-28 10:59:49,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 10:59:49,465 [I] connection success
2026-09-28 10:59:49,465 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 10:59:49,465 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 10:59:49,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 10:59:49,696 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 10:59:49,697 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:59:49,697 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 10:59:49,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:57173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 10:59:49,942 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 10:59:50.464830 4081 mini_ranger_kms.cc:310] Created kms service
I20260928 10:59:50.661109 4081 mini_ranger_kms.cc:326] Created kudu user
I20260928 10:59:50.702092 4081 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 10:59:50 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593187, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 10:59:51.098281 4081 mini_ranger_kms.cc:384] Added ranger policy
I20260928 10:59:51.098577 4081 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 10:59:51.098619 4081 mini_ranger_kms.cc:240] Using host: 127.3.252.84
I20260928 10:59:51.119743 4081 mini_ranger_kms.cc:276] Ranger KMS PID: 15337
I20260928 10:59:51.119908 4081 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.252.84:32837
Sep 28, 2026 10:59:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:59:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 10:59:53 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790593193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.252.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:53.224284 4081 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.338s user 0.118s sys 0.294s
I20260928 10:59:53.224479 4081 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 10:59:53.224695 4081 mini_ranger_kms.cc:401] 127.3.252.84:32837/kms/v1/keys
Sep 28 10:59:53 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.252.84@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
Sep 28 10:59:53 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.252.84@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
Sep 28 10:59:56 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 10:59:56.218024 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:43023
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:43023
--ranger_config_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:32837/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:59:56.548427 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:56.548723 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:56.548820 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:56.555953 15388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:59:56.556061 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:56.556104 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:56.556157 15388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:59:56.556203 15388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:59:56.566191 15388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.252.84:32837/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:43023
--ranger_config_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:56.567373 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:56.568655 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:56.579939 15396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:56.579973 15393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:56.579907 15394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:56.581102 15388 server_base.cc:1034] running on GCE node
I20260928 10:59:56.581799 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:56.583204 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:56.584429 15388 hybrid_clock.cc:648] HybridClock initialized: now 1790593196584377 us; error 36 us; skew 500 ppm
I20260928 10:59:56.584765 15388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:56 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593196, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:56.588099 15388 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 10:59:56.589584 15388 webserver.cc:460] Webserver started at http://127.3.252.126:42449/ using document root <none> and password file <none>
I20260928 10:59:56.590250 15388 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:56.590358 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:56.590646 15388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:59:56 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593196, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 10:59:56.648523 15388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "32abf67f67b240099b23a711f3ab9626"
format_stamp: "Formatted at 2026-09-28 10:59:56 on dist-test-slave-djjw"
server_key: "231913a874ea4a2eafbc32318de4515a"
server_key_iv: "50ecb78cff73940bfec30d5f3904f415"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:56.649720 15388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "32abf67f67b240099b23a711f3ab9626"
format_stamp: "Formatted at 2026-09-28 10:59:56 on dist-test-slave-djjw"
server_key: "231913a874ea4a2eafbc32318de4515a"
server_key_iv: "50ecb78cff73940bfec30d5f3904f415"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:56.656631 15388 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 10:59:56.674883 15403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:56.676367 15388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:59:56.676550 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "32abf67f67b240099b23a711f3ab9626"
format_stamp: "Formatted at 2026-09-28 10:59:56 on dist-test-slave-djjw"
server_key: "231913a874ea4a2eafbc32318de4515a"
server_key_iv: "50ecb78cff73940bfec30d5f3904f415"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:56.676766 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:56.693485 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:56.697382 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:56.697655 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:56.717468 15388 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:43023
I20260928 10:59:56.717468 15454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:43023 every 8 connection(s)
I20260928 10:59:56.718968 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 10:59:56.722817 15455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:56.723552 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 15388
I20260928 10:59:56.723742 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 10:59:56.724516 4081 external_mini_cluster.cc:1428] Setting key 093339825ec060048596181ba7ce7b70
I20260928 10:59:56.738471 15455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626: Bootstrap starting.
I20260928 10:59:56.742795 15455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:56.744179 15455 log.cc:824] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:56.747876 15455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626: No bootstrap required, opened a new log
Sep 28 10:59:56 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:56.756563 15455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32abf67f67b240099b23a711f3ab9626" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43023 } }
I20260928 10:59:56.757036 15455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:56.757135 15455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32abf67f67b240099b23a711f3ab9626, State: Initialized, Role: FOLLOWER
I20260928 10:59:56.757766 15455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32abf67f67b240099b23a711f3ab9626" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43023 } }
I20260928 10:59:56.758071 15455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:59:56.758203 15455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:59:56.758373 15455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:56.761480 15455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32abf67f67b240099b23a711f3ab9626" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43023 } }
I20260928 10:59:56.762055 15455 leader_election.cc:302] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32abf67f67b240099b23a711f3ab9626; no voters:
I20260928 10:59:56.762653 15455 leader_election.cc:288] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:59:56.762769 15460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:59:56.762795 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:56.727127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.3.252.126:43023)
0928 10:59:56.741756 (+ 14629us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:56.741773 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:56.741849 (+ 76us) server_negotiation.cc:407] Connection header received
0928 10:59:56.744225 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:56.744257 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:56.744806 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:56.745291 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:56.746749 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:56.748655 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:56.749951 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:56.751027 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:56.751202 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:56.755088 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:56.755097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:56.755113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:56.755157 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:56.758275 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:56.759111 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:56.759118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:56.759124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:56.759267 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:56.759641 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:56.759644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:56.759646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:56.760610 (+ 964us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:56.760814 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:56.761263 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 10:59:56.761605 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14375,"thread_start_us":14162,"threads_started":1}
I20260928 10:59:56.763834 15460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [term 1 LEADER]: Becoming Leader. State: Replica: 32abf67f67b240099b23a711f3ab9626, State: Running, Role: LEADER
I20260928 10:59:56.764473 15460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32abf67f67b240099b23a711f3ab9626" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43023 } }
I20260928 10:59:56.764957 15455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:59:56.767539 15461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32abf67f67b240099b23a711f3ab9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32abf67f67b240099b23a711f3ab9626" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43023 } } }
I20260928 10:59:56.768504 15461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [sys.catalog]: This master's current role is: LEADER
I20260928 10:59:56.767253 15462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32abf67f67b240099b23a711f3ab9626. Latest consensus state: current_term: 1 leader_uuid: "32abf67f67b240099b23a711f3ab9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32abf67f67b240099b23a711f3ab9626" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 43023 } } }
I20260928 10:59:56.768752 15462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626 [sys.catalog]: This master's current role is: LEADER
I20260928 10:59:56.781440 15469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:59:56.786657 15469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:59:56.787246 15455 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 10:59:56.804935 15469 catalog_manager.cc:1347] Generated new cluster ID: 95d6f3b543794e12b76cb579cbd1b76c
I20260928 10:59:56.805080 15469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:59:56.824460 15469 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:59:56.825816 15469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:59:56.834932 15469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626: Generated new TSK 0
I20260928 10:59:56.835976 15469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 10:59:57.611259 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:32837/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43023
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:59:57 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790593197, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 10:59:58.011469 15501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:58.011756 15501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:58.011895 15501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:58.019546 15501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:58.019667 15501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:58.019846 15501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 10:59:58.030287 15501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.252.84:32837/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:58.031687 15501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:58.033175 15501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:58.055604 15511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:58.055621 15512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:58.056507 15514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:58.057644 15501 server_base.cc:1034] running on GCE node
I20260928 10:59:58.058466 15501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:58.059819 15501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:58.061128 15501 hybrid_clock.cc:648] HybridClock initialized: now 1790593198061029 us; error 86 us; skew 500 ppm
I20260928 10:59:58.061509 15501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:58 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593198, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:58.065754 15501 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 10:59:58.068010 15501 webserver.cc:460] Webserver started at http://127.3.252.65:34073/ using document root <none> and password file <none>
I20260928 10:59:58.069064 15501 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:58.069217 15501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:58.069618 15501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:59:58 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593198, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 10:59:58.088795 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "cb08427759bf4487b94aab6fce445d5e"
format_stamp: "Formatted at 2026-09-28 10:59:58 on dist-test-slave-djjw"
server_key: "97d78b394fa12bc9694660d3452c211b"
server_key_iv: "02fe2d2d393c9971633ca647f7906307"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:58.089988 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "cb08427759bf4487b94aab6fce445d5e"
format_stamp: "Formatted at 2026-09-28 10:59:58 on dist-test-slave-djjw"
server_key: "97d78b394fa12bc9694660d3452c211b"
server_key_iv: "02fe2d2d393c9971633ca647f7906307"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:58.096460 15501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:59:58.138628 15521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:58.140805 15501 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.010s sys 0.005s
I20260928 10:59:58.141036 15501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "cb08427759bf4487b94aab6fce445d5e"
format_stamp: "Formatted at 2026-09-28 10:59:58 on dist-test-slave-djjw"
server_key: "97d78b394fa12bc9694660d3452c211b"
server_key_iv: "02fe2d2d393c9971633ca647f7906307"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:58.141345 15501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:58.160264 15501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:58.165012 15501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:58.165418 15501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:58.167843 15501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:58.169884 15501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:58.170018 15501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 10:59:58.170172 15501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:58.170260 15501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:58.233938 15501 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:44959
I20260928 10:59:58.234054 15633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:44959 every 8 connection(s)
I20260928 10:59:58.235287 15501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 10:59:58.241242 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 15501
I20260928 10:59:58.241421 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 10:59:58.242074 4081 external_mini_cluster.cc:1428] Setting key bdfda113658b01e3436c4af96f060b31
Sep 28 10:59:58 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593198, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:58.270272 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:58.238677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:39659 (local address 127.3.252.126:43023)
0928 10:59:58.239185 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:58.239191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:58.240225 (+ 1034us) server_negotiation.cc:407] Connection header received
0928 10:59:58.251200 (+ 10975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:58.251206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:58.251307 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:58.251413 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:58.253601 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:58.254626 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:58.257582 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:58.258252 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:58.258354 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:58.262859 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:58.262867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:58.262872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:58.262911 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:58.266532 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:58.267458 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:58.267464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:58.267467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:58.267618 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:58.268158 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:58.268163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:58.268166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:58.268521 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:58.268651 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:58.269643 (+ 992us) server_negotiation.cc:299] Negotiation successful
0928 10:59:58.269906 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260928 10:59:58.273005 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:58.239114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:43023 (local address 127.3.252.65:39659)
0928 10:59:58.239989 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:58.240070 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:58.242614 (+ 2544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:58.251945 (+ 9331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:58.251959 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:58.252575 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:58.253398 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:58.253426 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:58.254817 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:58.254833 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:58.257410 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:58.257426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:58.258713 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:58.258720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:58.259021 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:58.259775 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:58.259805 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:58.262576 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:58.266694 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:58.266705 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:58.266723 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:58.267315 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:58.267755 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:58.267760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:58.267764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:58.268020 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:58.268739 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:58.268747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:58.269184 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 10:59:58.270929 (+ 1745us) client_negotiation.cc:240] Negotiation successful
0928 10:59:58.271496 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
I20260928 10:59:58.275022 15634 heartbeater.cc:347] Connected to a master server at 127.3.252.126:43023
I20260928 10:59:58.275359 15634 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:58.276192 15634 heartbeater.cc:511] Master 127.3.252.126:43023 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 10:59:58.278681 15420 ts_manager.cc:194] Registered new tserver with Master: cb08427759bf4487b94aab6fce445d5e (127.3.252.65:44959)
I20260928 10:59:58.280715 15420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:39659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 10:59:58.320961 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:32837/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43023
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:59:58 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593197, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
Sep 28 10:59:58 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593197, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
W20260928 10:59:58.805923 15640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:58.806406 15640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:58.806599 15640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:58.818086 15640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:58.818238 15640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:58.818480 15640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 10:59:58.834540 15640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.252.84:32837/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:58.836457 15640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:58.838277 15640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:58.876149 15647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:58.877978 15648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:58.895537 15650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:58.896416 15640 server_base.cc:1034] running on GCE node
I20260928 10:59:58.897269 15640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:58.899968 15640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:58.901291 15640 hybrid_clock.cc:648] HybridClock initialized: now 1790593198901168 us; error 100 us; skew 500 ppm
I20260928 10:59:58.906265 15640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:58 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593198, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:58.910452 15640 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 10:59:58.924075 15640 webserver.cc:460] Webserver started at http://127.3.252.66:43597/ using document root <none> and password file <none>
I20260928 10:59:58.924923 15640 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:58.925034 15640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:58.925345 15640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:59:58 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593198, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 10:59:58.941994 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "92df14ede7c54a7483bdac67b01d16c2"
format_stamp: "Formatted at 2026-09-28 10:59:58 on dist-test-slave-djjw"
server_key: "4722ed67cd9a382e1197250cc72ab72c"
server_key_iv: "f250a6d7399ef35ad84894bab946932c"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:58.942857 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "92df14ede7c54a7483bdac67b01d16c2"
format_stamp: "Formatted at 2026-09-28 10:59:58 on dist-test-slave-djjw"
server_key: "4722ed67cd9a382e1197250cc72ab72c"
server_key_iv: "f250a6d7399ef35ad84894bab946932c"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:58.948256 15640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:59:58.964315 15664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:58.965973 15640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:59:58.966142 15640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "92df14ede7c54a7483bdac67b01d16c2"
format_stamp: "Formatted at 2026-09-28 10:59:58 on dist-test-slave-djjw"
server_key: "4722ed67cd9a382e1197250cc72ab72c"
server_key_iv: "f250a6d7399ef35ad84894bab946932c"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:58.966360 15640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:58.991802 15640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:58.996433 15640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:58.996726 15640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:58.997738 15640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:58.999514 15640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:58.999601 15640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:58.999722 15640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:58.999773 15640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:59.030274 15640 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:44619
I20260928 10:59:59.030313 15776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:44619 every 8 connection(s)
I20260928 10:59:59.031512 15640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 10:59:59.038802 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 15640
I20260928 10:59:59.038957 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 10:59:59.039386 4081 external_mini_cluster.cc:1428] Setting key 6d08c74de7b012043bbd0f26ed009d06
Sep 28 10:59:59 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593198, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:59.051956 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.033955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:46399 (local address 127.3.252.126:43023)
0928 10:59:59.034404 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.034409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.035262 (+ 853us) server_negotiation.cc:407] Connection header received
0928 10:59:59.037063 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.037069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.037158 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.037250 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:59.038755 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.039628 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.041893 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.042292 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.042403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.046033 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:59.046038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:59.046041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:59.046075 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:59.048955 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.049625 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.049631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.049635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.049776 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.050233 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.050238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.050241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.050602 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:59.050743 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.051392 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.051598 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260928 10:59:59.052742 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.034389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:43023 (local address 127.3.252.66:46399)
0928 10:59:59.035102 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.035145 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:59.036800 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:59.037495 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:59.037504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:59.037985 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:59.038590 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.038608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.039789 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.039792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.041730 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.041744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.042509 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.042517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.042731 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.043360 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:59.043388 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:59.045788 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:59.049079 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:59.049086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:59.049100 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:59.049509 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:59.049938 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:59.049941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:59.049944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:59.050136 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:59.050710 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:59.050715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:59.051055 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 10:59:59.051272 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 10:59:59.051648 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":294,"threads_started":1}
I20260928 10:59:59.054196 15777 heartbeater.cc:347] Connected to a master server at 127.3.252.126:43023
I20260928 10:59:59.054517 15777 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:59.055291 15777 heartbeater.cc:511] Master 127.3.252.126:43023 requested a full tablet report, sending...
I20260928 10:59:59.056795 15420 ts_manager.cc:194] Registered new tserver with Master: 92df14ede7c54a7483bdac67b01d16c2 (127.3.252.66:44619)
I20260928 10:59:59.057623 15420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:46399
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 10:59:59.089277 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:32837/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43023
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:59:59.284582 15634 heartbeater.cc:503] Master 127.3.252.126:43023 was elected leader, sending a full tablet report...
W20260928 10:59:59.420612 15783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:59:59.420907 15783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:59:59.421026 15783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:59:59.428486 15783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:59:59.428614 15783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:59:59.428758 15783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 10:59:59.438344 15783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.252.84:32837/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:59:59.439548 15783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:59:59.440768 15783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:59:59.453665 15788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:59.453601 15791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:59:59.453866 15789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:59:59.454270 15783 server_base.cc:1034] running on GCE node
I20260928 10:59:59.454844 15783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:59:59.455731 15783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:59:59.456923 15783 hybrid_clock.cc:648] HybridClock initialized: now 1790593199456868 us; error 48 us; skew 500 ppm
I20260928 10:59:59.457213 15783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:59:59 dist-test-slave-djjw krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593199, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:59:59.460654 15783 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 10:59:59.462116 15783 webserver.cc:460] Webserver started at http://127.3.252.67:34015/ using document root <none> and password file <none>
I20260928 10:59:59.462766 15783 fs_manager.cc:359] Metadata directory not provided
I20260928 10:59:59.462869 15783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:59:59.463150 15783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:59:59 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593199, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 10:59:59.476797 15783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5"
format_stamp: "Formatted at 2026-09-28 10:59:59 on dist-test-slave-djjw"
server_key: "9a83dd3d48fbb911b1ea3b087884a3d2"
server_key_iv: "f57961f78137fe1be05bed2e056d2ce8"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:59.477591 15783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5"
format_stamp: "Formatted at 2026-09-28 10:59:59 on dist-test-slave-djjw"
server_key: "9a83dd3d48fbb911b1ea3b087884a3d2"
server_key_iv: "f57961f78137fe1be05bed2e056d2ce8"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:59.482935 15783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:59:59.499104 15798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:59.500664 15783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:59:59.500829 15783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5"
format_stamp: "Formatted at 2026-09-28 10:59:59 on dist-test-slave-djjw"
server_key: "9a83dd3d48fbb911b1ea3b087884a3d2"
server_key_iv: "f57961f78137fe1be05bed2e056d2ce8"
server_key_version: "kuduclusterkey@0"
I20260928 10:59:59.501039 15783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:59:59.532501 15783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:59:59.536454 15783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:59:59.536728 15783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:59:59.537664 15783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:59:59.539022 15783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:59:59.539090 15783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:59.539202 15783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:59:59.539249 15783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:59:59.570804 15783 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:38413
I20260928 10:59:59.571292 15910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:38413 every 8 connection(s)
I20260928 10:59:59.572284 15783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 10:59:59.580691 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 15783
I20260928 10:59:59.580854 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 10:59:59.581329 4081 external_mini_cluster.cc:1428] Setting key b0a9f71762d1933b9bc0112252ae89f8
Sep 28 10:59:59 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593199, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 10:59:59.592281 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.574801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:56777 (local address 127.3.252.126:43023)
0928 10:59:59.575222 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.575228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.576149 (+ 921us) server_negotiation.cc:407] Connection header received
0928 10:59:59.577859 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.577865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.577958 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.578054 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:59.579507 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.580343 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.582327 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.582830 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.582931 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.586428 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:59.586434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:59.586437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:59.586472 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:59.589346 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.590129 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.590134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.590136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.590284 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.590738 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.590742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.590744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.591064 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:59.591186 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.591854 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.592025 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":202,"threads_started":1}
I20260928 10:59:59.593251 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.575320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:43023 (local address 127.3.252.67:56777)
0928 10:59:59.575989 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.576033 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:59.577579 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:59.578311 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:59.578319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:59.578738 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:59:59.579333 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.579352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.580498 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.580501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.582189 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.582202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.583066 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.583072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.583384 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.584044 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:59:59.584076 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:59:59.586223 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:59:59.589474 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:59.589481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:59.589495 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:59.589997 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:59.590394 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:59:59.590397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:59:59.590399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:59:59.590596 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:59:59.591190 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:59:59.591196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:59:59.591513 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 10:59:59.591744 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 10:59:59.592169 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":259,"threads_started":1}
I20260928 10:59:59.594767 15911 heartbeater.cc:347] Connected to a master server at 127.3.252.126:43023
I20260928 10:59:59.595083 15911 heartbeater.cc:464] Registering TS with master...
I20260928 10:59:59.595937 15911 heartbeater.cc:511] Master 127.3.252.126:43023 requested a full tablet report, sending...
I20260928 10:59:59.597357 15420 ts_manager.cc:194] Registered new tserver with Master: 4ec34b7c814a4a21a7bf9dc613ebccd5 (127.3.252.67:38413)
I20260928 10:59:59.598169 15420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:56777
I20260928 10:59:59.608975 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:59:59.625414 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.613482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.3.252.126:43023)
0928 10:59:59.613679 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.613686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.614071 (+ 385us) server_negotiation.cc:407] Connection header received
0928 10:59:59.614252 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.614256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.614329 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.614407 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:59.615728 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.616417 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.617671 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.618132 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.618235 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.620576 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:59.620580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:59.620584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:59.620614 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:59.622952 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.623591 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.623595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.623597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.623719 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.624006 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.624008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.624010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.624265 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:59.624847 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.624946 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.625162 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:59:59.629185 15420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:59:59.635989 15420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:59:59.659189 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.651244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.3.252.65:44959)
0928 10:59:59.651914 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.651921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.651947 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:59:59.652034 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.652040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.652252 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.653550 (+ 1298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:59.653780 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.654823 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.656892 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.657799 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.658363 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.658609 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.658682 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.658851 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":346,"threads_started":1}
I20260928 10:59:59.659224 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.648962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.3.252.67:38413)
0928 10:59:59.649538 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.649545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.650425 (+ 880us) server_negotiation.cc:407] Connection header received
0928 10:59:59.650752 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.650759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.650999 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.651156 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:59.652109 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.653438 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.656402 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.657884 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.658003 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.658247 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.658766 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.658927 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":684,"threads_started":1}
I20260928 10:59:59.659212 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.650848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57156)
0928 10:59:59.651292 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.651319 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:59.651422 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:59.652463 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:59.652467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:59.652492 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:59.653355 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.653369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.654959 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.654968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.656735 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.656749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.658077 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.658084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.658347 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.658466 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 10:59:59.658592 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 10:59:59.658834 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":113,"thread_start_us":584,"threads_started":1}
I20260928 10:59:59.659255 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.648515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.3.252.66:44619)
0928 10:59:59.649123 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.649130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.650490 (+ 1360us) server_negotiation.cc:407] Connection header received
0928 10:59:59.650727 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.650733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.650992 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.652007 (+ 1015us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:59.652226 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.653664 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.656400 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.657879 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.658115 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.658280 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.658766 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.658917 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":348,"threads_started":1}
I20260928 10:59:59.659940 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.648364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:38413 (local address 127.0.0.1:48322)
0928 10:59:59.650288 (+ 1924us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.650329 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:59.650506 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:59.651467 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:59.651471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:59.651495 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:59.651921 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.651933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.653574 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.653577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.656230 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.656239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.658359 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.658365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.658572 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.658604 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:59:59.659402 (+ 798us) client_negotiation.cc:240] Negotiation successful
0928 10:59:59.659537 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1701,"mutex_wait_us":125,"spinlock_wait_cycles":18816,"thread_start_us":2141,"threads_started":1}
I20260928 10:59:59.660099 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.649556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:44619 (local address 127.0.0.1:41072)
0928 10:59:59.650285 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.650325 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:59.650507 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:59.651229 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:59.651237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:59.651272 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:59.651741 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.651754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.653802 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.653807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.656220 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.656236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.658135 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.658141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.658390 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.658466 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 10:59:59.658592 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 10:59:59.659697 (+ 1105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":1044,"spinlock_wait_cycles":3968,"thread_start_us":350,"threads_started":1}
I20260928 10:59:59.663827 15846 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb133d3f6a44bc7a228734e566e0e80 (DEFAULT_TABLE table=test-table [id=69c8af2f38eb4bd29db1d6d0c9d25ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:59:59.664023 15569 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb133d3f6a44bc7a228734e566e0e80 (DEFAULT_TABLE table=test-table [id=69c8af2f38eb4bd29db1d6d0c9d25ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:59:59.664023 15712 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb133d3f6a44bc7a228734e566e0e80 (DEFAULT_TABLE table=test-table [id=69c8af2f38eb4bd29db1d6d0c9d25ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:59:59.665140 15846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb133d3f6a44bc7a228734e566e0e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:59.665305 15712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb133d3f6a44bc7a228734e566e0e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:59.665315 15569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb133d3f6a44bc7a228734e566e0e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:59:59.676911 15930 tablet_bootstrap.cc:492] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Bootstrap starting.
I20260928 10:59:59.679111 15931 tablet_bootstrap.cc:492] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Bootstrap starting.
I20260928 10:59:59.681720 15930 tablet_bootstrap.cc:654] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:59.683077 15931 tablet_bootstrap.cc:654] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:59.683029 15930 log.cc:824] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:59.684377 15931 log.cc:824] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:59.685739 15932 tablet_bootstrap.cc:492] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: Bootstrap starting.
I20260928 10:59:59.686236 15930 tablet_bootstrap.cc:492] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: No bootstrap required, opened a new log
I20260928 10:59:59.686507 15930 ts_tablet_manager.cc:1397] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 10:59:59.687409 15931 tablet_bootstrap.cc:492] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: No bootstrap required, opened a new log
I20260928 10:59:59.688151 15931 ts_tablet_manager.cc:1397] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 10:59:59.688694 15932 tablet_bootstrap.cc:654] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: Neither blocks nor log segments found. Creating new log.
I20260928 10:59:59.689711 15932 log.cc:824] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:59:59.692070 15932 tablet_bootstrap.cc:492] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: No bootstrap required, opened a new log
I20260928 10:59:59.692297 15932 ts_tablet_manager.cc:1397] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 10:59:59.697693 15930 raft_consensus.cc:348] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.698244 15930 raft_consensus.cc:374] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:59.698356 15930 raft_consensus.cc:729] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec34b7c814a4a21a7bf9dc613ebccd5, State: Initialized, Role: FOLLOWER
I20260928 10:59:59.698256 15931 raft_consensus.cc:348] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.698778 15931 raft_consensus.cc:374] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:59.698894 15931 raft_consensus.cc:729] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92df14ede7c54a7483bdac67b01d16c2, State: Initialized, Role: FOLLOWER
I20260928 10:59:59.699025 15930 consensus_queue.cc:260] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.699453 15931 consensus_queue.cc:260] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.700223 15911 heartbeater.cc:503] Master 127.3.252.126:43023 was elected leader, sending a full tablet report...
I20260928 10:59:59.700217 15932 raft_consensus.cc:348] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.700606 15932 raft_consensus.cc:374] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:59:59.700713 15932 raft_consensus.cc:729] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb08427759bf4487b94aab6fce445d5e, State: Initialized, Role: FOLLOWER
I20260928 10:59:59.700917 15930 ts_tablet_manager.cc:1428] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260928 10:59:59.701318 15932 consensus_queue.cc:260] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.703788 15932 ts_tablet_manager.cc:1428] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: Time spent starting tablet: real 0.011s user 0.011s sys 0.002s
I20260928 10:59:59.704672 15777 heartbeater.cc:503] Master 127.3.252.126:43023 was elected leader, sending a full tablet report...
I20260928 10:59:59.706350 15939 raft_consensus.cc:482] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:59:59.706835 15939 raft_consensus.cc:504] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.709317 15939 leader_election.cc:288] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec34b7c814a4a21a7bf9dc613ebccd5 (127.3.252.67:38413), cb08427759bf4487b94aab6fce445d5e (127.3.252.65:44959)
I20260928 10:59:59.710218 15931 ts_tablet_manager.cc:1428] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Time spent starting tablet: real 0.022s user 0.011s sys 0.009s
I20260928 10:59:59.717053 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.709696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:52969 (local address 127.3.252.67:38413)
0928 10:59:59.709889 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.709895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.710347 (+ 452us) server_negotiation.cc:407] Connection header received
0928 10:59:59.710561 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.710567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.710632 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.710709 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:59.711610 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.712660 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.714948 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.716138 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.716320 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.716386 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.716756 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.716862 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 10:59:59.717316 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.709604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:38413 (local address 127.3.252.66:52969)
0928 10:59:59.710262 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.710277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:59.710449 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:59.710865 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:59.710870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:59.710893 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:59.711435 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.711450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.712766 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.712771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.714800 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.714812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.716402 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.716408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.716574 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.716608 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:59:59.716712 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 10:59:59.716840 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":376,"threads_started":1}
I20260928 10:59:59.718358 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.710410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.3.252.66:33577)
0928 10:59:59.711572 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.711587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:59:59.711789 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:59:59.712230 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:59:59.712234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:59:59.712257 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:59:59.712560 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.712570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.713959 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.713963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.716206 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:59:59.716219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.717769 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.717776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:59:59.717961 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.717983 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 10:59:59.718059 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 10:59:59.718142 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":908,"threads_started":1}
I20260928 10:59:59.718264 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb133d3f6a44bc7a228734e566e0e80" candidate_uuid: "92df14ede7c54a7483bdac67b01d16c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" is_pre_election: true
I20260928 10:59:59.718837 15866 raft_consensus.cc:2454] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92df14ede7c54a7483bdac67b01d16c2 in term 0.
I20260928 10:59:59.719465 15666 leader_election.cc:302] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec34b7c814a4a21a7bf9dc613ebccd5, 92df14ede7c54a7483bdac67b01d16c2; no voters:
I20260928 10:59:59.719856 15939 raft_consensus.cc:2781] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:59:59.720002 15939 raft_consensus.cc:482] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:59:59.720093 15939 raft_consensus.cc:3037] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:59.722546 15939 raft_consensus.cc:504] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.723090 15939 leader_election.cc:288] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4ec34b7c814a4a21a7bf9dc613ebccd5 (127.3.252.67:38413), cb08427759bf4487b94aab6fce445d5e (127.3.252.65:44959)
I20260928 10:59:59.723605 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb133d3f6a44bc7a228734e566e0e80" candidate_uuid: "92df14ede7c54a7483bdac67b01d16c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5"
I20260928 10:59:59.723788 15866 raft_consensus.cc:3037] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:59.726320 15866 raft_consensus.cc:2454] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92df14ede7c54a7483bdac67b01d16c2 in term 1.
I20260928 10:59:59.726677 15666 leader_election.cc:302] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec34b7c814a4a21a7bf9dc613ebccd5, 92df14ede7c54a7483bdac67b01d16c2; no voters:
I20260928 10:59:59.726926 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.710832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:33577 (local address 127.3.252.65:44959)
0928 10:59:59.711033 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.711038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.711652 (+ 614us) server_negotiation.cc:407] Connection header received
0928 10:59:59.711936 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.711942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.712009 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.712066 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:59:59.712716 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.713795 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.716360 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.717580 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.717725 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.717787 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.726535 (+ 8748us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.726695 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:59:59.726958 15939 raft_consensus.cc:2781] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:59:59.727095 15939 raft_consensus.cc:686] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 1 LEADER]: Becoming Leader. State: Replica: 92df14ede7c54a7483bdac67b01d16c2, State: Running, Role: LEADER
I20260928 10:59:59.727607 15939 consensus_queue.cc:237] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } }
I20260928 10:59:59.727947 15589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb133d3f6a44bc7a228734e566e0e80" candidate_uuid: "92df14ede7c54a7483bdac67b01d16c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb08427759bf4487b94aab6fce445d5e" is_pre_election: true
I20260928 10:59:59.728403 15589 raft_consensus.cc:2454] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92df14ede7c54a7483bdac67b01d16c2 in term 0.
I20260928 10:59:59.728866 15588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb133d3f6a44bc7a228734e566e0e80" candidate_uuid: "92df14ede7c54a7483bdac67b01d16c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb08427759bf4487b94aab6fce445d5e"
I20260928 10:59:59.729074 15588 raft_consensus.cc:3037] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:59:59.731345 15588 raft_consensus.cc:2454] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92df14ede7c54a7483bdac67b01d16c2 in term 1.
I20260928 10:59:59.734690 15420 catalog_manager.cc:5495] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92df14ede7c54a7483bdac67b01d16c2 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "92df14ede7c54a7483bdac67b01d16c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92df14ede7c54a7483bdac67b01d16c2" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 } health_report { overall_health: UNKNOWN } } }
W20260928 10:59:59.738400 15635 tablet.cc:2369] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:59:59.769367 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.763662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.3.252.66:44619)
0928 10:59:59.763851 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.763855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.763873 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:59:59.764016 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.764020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.764102 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.764225 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:59:59.764545 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.765367 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.766893 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.767737 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.767864 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.768316 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:59:59.768877 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:59:59.768983 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.769093 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.769158 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:59:59.777631 15588 raft_consensus.cc:1261] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 1 FOLLOWER]: Refusing update from remote peer 92df14ede7c54a7483bdac67b01d16c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:59:59.777648 15866 raft_consensus.cc:1261] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 1 FOLLOWER]: Refusing update from remote peer 92df14ede7c54a7483bdac67b01d16c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:59:59.778558 15939 consensus_queue.cc:1035] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb08427759bf4487b94aab6fce445d5e" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:59:59.778851 15937 consensus_queue.cc:1035] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec34b7c814a4a21a7bf9dc613ebccd5" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:59:59.783474 15778 tablet.cc:2369] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:59:59.802569 15418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:59:59.802824 15418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:59:59.806232 15418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32abf67f67b240099b23a711f3ab9626: Sending DeleteTablet for 3 replicas of tablet 5fb133d3f6a44bc7a228734e566e0e80
I20260928 10:59:59.807404 15712 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb133d3f6a44bc7a228734e566e0e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:59:59 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:41072
I20260928 10:59:59.807454 15569 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb133d3f6a44bc7a228734e566e0e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:59:59 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:57156
I20260928 10:59:59.807559 15846 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb133d3f6a44bc7a228734e566e0e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:59:59 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:48322
I20260928 10:59:59.809226 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 15501
I20260928 10:59:59.823376 15954 tablet_replica.cc:330] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: stopping tablet replica
I20260928 10:59:59.824599 15954 raft_consensus.cc:2229] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:59:59.826069 15954 raft_consensus.cc:2258] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:59:59.829727 15954 ts_tablet_manager.cc:1905] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:59:59.830587 15955 tablet_replica.cc:330] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: stopping tablet replica
Sep 28 10:59:59 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 10:59:59.831707 15955 raft_consensus.cc:2229] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:59:59.831892 15953 tablet_replica.cc:330] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: stopping tablet replica
I20260928 10:59:59.832350 15953 raft_consensus.cc:2229] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:59:59.832427 15955 raft_consensus.cc:2258] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:59:59.834411 15953 raft_consensus.cc:2258] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:59:59.834534 15955 ts_tablet_manager.cc:1905] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:59:59.836670 15953 ts_tablet_manager.cc:1905] T 5fb133d3f6a44bc7a228734e566e0e80 P cb08427759bf4487b94aab6fce445d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:59:59.838709 15954 ts_tablet_manager.cc:1918] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:59:59.838986 15954 log.cc:1196] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/5fb133d3f6a44bc7a228734e566e0e80
I20260928 10:59:59.839658 15954 ts_tablet_manager.cc:1939] T 5fb133d3f6a44bc7a228734e566e0e80 P 92df14ede7c54a7483bdac67b01d16c2: Deleting consensus metadata
I20260928 10:59:59.841485 15955 ts_tablet_manager.cc:1918] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:59:59.841745 15955 log.cc:1196] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/5fb133d3f6a44bc7a228734e566e0e80
I20260928 10:59:59.842235 15407 catalog_manager.cc:4841] TS 92df14ede7c54a7483bdac67b01d16c2 (127.3.252.66:44619): tablet 5fb133d3f6a44bc7a228734e566e0e80 (table test-table [id=69c8af2f38eb4bd29db1d6d0c9d25ed4]) successfully deleted
I20260928 10:59:59.842434 15955 ts_tablet_manager.cc:1939] T 5fb133d3f6a44bc7a228734e566e0e80 P 4ec34b7c814a4a21a7bf9dc613ebccd5: Deleting consensus metadata
I20260928 10:59:59.844200 15405 catalog_manager.cc:4841] TS 4ec34b7c814a4a21a7bf9dc613ebccd5 (127.3.252.67:38413): tablet 5fb133d3f6a44bc7a228734e566e0e80 (table test-table [id=69c8af2f38eb4bd29db1d6d0c9d25ed4]) successfully deleted
I20260928 10:59:59.844774 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.810792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.3.252.65:44959)
0928 10:59:59.810977 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.810984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.816237 (+ 5253us) server_negotiation.cc:407] Connection header received
0928 10:59:59.817150 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.817157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.817275 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.817351 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:59.820996 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.822308 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.825846 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.826679 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.826882 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.837037 (+ 10155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:59.837055 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:59.837072 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:59.837120 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:59.840897 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.841984 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.841994 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.841998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.842169 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.842675 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.842680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.842683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.843203 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:59.843493 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.844109 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.844354 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:59:59.846208 15629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:59:59.943296 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 15501
I20260928 10:59:59.957327 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 15640
W20260928 10:59:59.957343 15405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.252.65:44959 (error 108)
I20260928 10:59:59.960680 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.960213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57178)
0928 10:59:59.960389 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 10:59:59.960512 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 10:59:59.961011 15405 catalog_manager.cc:4584] TS cb08427759bf4487b94aab6fce445d5e (127.3.252.65:44959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fb133d3f6a44bc7a228734e566e0e80: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Sep 28 10:59:59 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 10:59:59.974264 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:59:59.958400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.3.252.66:44619)
0928 10:59:59.958634 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:59:59.958641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:59:59.958841 (+ 200us) server_negotiation.cc:407] Connection header received
0928 10:59:59.959052 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:59:59.959084 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:59:59.959217 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:59:59.960069 (+ 852us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:59:59.960788 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.961709 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.963075 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:59:59.963500 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:59:59.963615 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:59:59.968187 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:59:59.968202 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:59:59.968217 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:59:59.968262 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:59:59.971314 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.972161 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.972168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.972170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.972329 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:59:59.972719 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:59:59.972722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:59:59.972724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:59:59.973064 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:59:59.973308 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:59:59.973649 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 10:59:59.973812 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:59:59.974956 15772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:00.014914 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.014369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57188)
0928 11:00:00.014589 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:00.014711 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 11:00:00.015911 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.015542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57192)
0928 11:00:00.015676 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:00.015772 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 11:00:00.076334 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.075835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57196)
0928 11:00:00.076058 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:00.076147 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 11:00:00.077279 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.076967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57212)
0928 11:00:00.077089 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:00.077156 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 11:00:00.089197 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 15640
I20260928 11:00:00.104317 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 15783
Sep 28 11:00:00 dist-test-slave-djjw krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 11:00:00.119304 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.105128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.3.252.67:38413)
0928 11:00:00.105333 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:00.105339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:00.105359 (+ 20us) server_negotiation.cc:407] Connection header received
0928 11:00:00.105546 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:00.105551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:00.105616 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:00.105698 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:00.107033 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:00.107746 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:00.109041 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:00.109426 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:00.109530 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:00.113988 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:00.114009 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:00.114030 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:00.114071 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:00.116506 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:00.117195 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:00.117203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:00.117205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:00.117339 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:00.117703 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:00.117707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:00.117709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:00.118122 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:00.118407 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:00.118704 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 11:00:00.118923 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 11:00:00.120234 15906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:00.158046 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.157450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57216)
0928 11:00:00.157701 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:00.157812 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 11:00:00.159111 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.158698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57226)
0928 11:00:00.158901 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:00.158984 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 11:00:00.211381 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 15783
I20260928 11:00:00.224571 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 15388
I20260928 11:00:00.225713 15450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:00.336973 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:00.336164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:44959 (local address 127.0.0.1:57242)
0928 11:00:00.336360 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:00.336459 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:44959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 11:00:00.337064 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 15388
I20260928 11:00:00.356846 4081 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 11:00:00.692314 4081 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 11:00:00.692528 4081 mini_ranger.cc:66] Stopping Ranger...
I20260928 11:00:01.027887 4081 mini_ranger.cc:68] Stopped Ranger
2026-09-28 11:00:01.028 UTC [14974] LOG: received smart shutdown request
2026-09-28 11:00:01.031 UTC [14974] LOG: background worker "logical replication launcher" (PID 14983) exited with exit code 1
2026-09-28 11:00:01.032 UTC [14978] LOG: shutting down
2026-09-28 11:00:01.194 UTC [14974] LOG: database system is shut down
2026-09-28T11:00:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47701 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:00:01 dist-test-slave-djjw krb5kdc[15970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:00:01 dist-test-slave-djjw krb5kdc[15970](info): set up 2 sockets
Sep 28 11:00:01 dist-test-slave-djjw krb5kdc[15970](info): commencing operation
krb5kdc: starting...
W20260928 11:00:02.322193 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:00:02 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:00:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/rangeradmin_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/rangeradmin_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/rangerlookup_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/rangerlookup_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
I20260928 11:00:02.632310 4081 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktiZmwa/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/postgres -l logfile start
2026-09-28 11:00:05.083 UTC [16000] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 11:00:05.083 UTC [16000] LOG: listening on IPv4 address "127.3.252.84", port 33581
2026-09-28 11:00:05.086 UTC [16000] LOG: listening on Unix socket "/tmp/.s.PGSQL.33581"
2026-09-28 11:00:05.107 UTC [16003] LOG: database system was shut down at 2026-09-28 11:00:03 UTC
2026-09-28 11:00:05.111 UTC [16000] LOG: database system is ready to accept connections
I20260928 11:00:06.005860 4081 mini_postgres.cc:96] Postgres bound to 33581
2026-09-28 11:00:06.022 UTC [16011] FATAL: database "slave" does not exist
127.3.252.84:33581 - accepting connections
I20260928 11:00:06.022750 4081 mini_ranger.cc:161] Starting Ranger...
I20260928 11:00:06.046314 4081 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 11:00:06.511193 4081 mini_ranger.cc:87] Created ranger Postgres database
I20260928 11:00:06.511291 4081 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-admin
2026-09-28 11:00:06,796 [I] DB FLAVOR :POSTGRES
2026-09-28 11:00:06,796 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 11:00:06,796 [I] Checking connection..
2026-09-28 11:00:06,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 11:00:07,028 [I] Checking connection passed.
2026-09-28 11:00:07,029 [I] --------- Verifying version history table ---------
2026-09-28 11:00:07,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 11:00:07,269 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 11:00:07,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 11:00:07,513 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 11:00:07,513 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 11:00:07,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 11:00:07.736 UTC [16075] WARNING: there is no transaction in progress
2026-09-28 11:00:07,742 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 11:00:07,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 11:00:07,984 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 11:00:07,984 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 11:00:07,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 11:00:08,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 11:00:08,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-djjw.c.gcp-upstream.internal','N') ;"
2026-09-28 11:00:08,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 11:00:08,965 [I] Table x_portal_user does not exist in database ranger
2026-09-28 11:00:08,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 11:00:09,227 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 11:00:09,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 11:00:09,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 11:00:09,725 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 11:00:09,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 11:00:10.665 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.680 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.695 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.709 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.729 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.744 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.796 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.802 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.815 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.824 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.834 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.847 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.856 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.862 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.868 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:10.874 UTC [16183] WARNING: there is no transaction in progress
2026-09-28 11:00:11,470 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 11:00:11,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 11:00:11,706 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 11:00:11,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 11:00:11,960 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 11:00:11,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 11:00:12,215 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 11:00:12,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 11:00:12,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 11:00:12,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-djjw.c.gcp-upstream.internal';"
2026-09-28 11:00:12,910 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 11:00:12,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 11:00:13,144 [I] DB_PATCHES have already been applied
I20260928 11:00:13.150348 4081 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktiZmwa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 11:00:13.150439 4081 mini_ranger.cc:193] Using host: 127.3.252.84
I20260928 11:00:13.163709 4081 mini_ranger.cc:227] Ranger admin URL: http://127.3.252.84:35835
Sep 28, 2026 11:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:00:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:00:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 11:00:14.220494 4081 mini_ranger.cc:160] Time spent starting Ranger: real 8.198s user 0.003s sys 0.034s
Sep 28, 2026 11:00:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:00:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 11:00:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:00:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:00:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 11:00:35.594518 4081 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.252.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.252.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/rangerkms_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.252.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/rangerkms_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.252.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.252.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.252.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/HTTP_127.3.252.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 11:00:35 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 11:00:35.746071 4081 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 11:00:35.764850 4081 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 11:00:36.621337 4081 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 11:00:36.621464 4081 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms
I20260928 11:00:36.621503 4081 mini_ranger_kms.cc:227] Using postgres at 127.3.252.84:33581
2026-09-28 11:00:36,946 [I] DB FLAVOR :POSTGRES
2026-09-28 11:00:36,946 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 11:00:36,946 [I] Checking connection
2026-09-28 11:00:36,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 11:00:37,177 [I] connection success
2026-09-28 11:00:37,178 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 11:00:37,178 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 11:00:37,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 11:00:37,405 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 11:00:37,405 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 11:00:37,405 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 11:00:37,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.252.84:33581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 11:00:37,658 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 11:00:38.140794 4081 mini_ranger_kms.cc:310] Created kms service
I20260928 11:00:38.342497 4081 mini_ranger_kms.cc:326] Created kudu user
I20260928 11:00:38.388017 4081 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 11:00:38 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:38.782703 4081 mini_ranger_kms.cc:384] Added ranger policy
I20260928 11:00:38.783028 4081 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktiZmwa/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktiZmwa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 11:00:38.783084 4081 mini_ranger_kms.cc:240] Using host: 127.3.252.84
I20260928 11:00:38.803295 4081 mini_ranger_kms.cc:276] Ranger KMS PID: 16363
I20260928 11:00:38.803421 4081 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.252.84:41655
Sep 28, 2026 11:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 11:00:40.811793 4081 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.066s user 0.067s sys 0.270s
I20260928 11:00:40.812002 4081 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 11:00:40.812151 4081 mini_ranger_kms.cc:401] 127.3.252.84:41655/kms/v1/keys
Sep 28 11:00:41 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790593241, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.252.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 11:00:41 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593241, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.252.84@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
Sep 28 11:00:41 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593241, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.252.84@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
Sep 28 11:00:44 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:00:44.158596 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:42787
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:42787
--ranger_config_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:00:44.478890 16414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:44.479220 16414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:44.479342 16414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:44.486640 16414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:00:44.486760 16414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:44.486815 16414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:44.486889 16414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:00:44.486948 16414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:00:44.497150 16414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.252.84:41655/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:42787
--ranger_config_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:44.498525 16414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:44.499877 16414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:44.511626 16419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:44.511699 16422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:44.511592 16420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:44.511654 16414 server_base.cc:1034] running on GCE node
I20260928 11:00:44.512667 16414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:44.513965 16414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:44.515154 16414 hybrid_clock.cc:648] HybridClock initialized: now 1790593244515116 us; error 33 us; skew 500 ppm
I20260928 11:00:44.515450 16414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:44 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593244, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:44.522182 16414 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:00:44.523636 16414 webserver.cc:460] Webserver started at http://127.3.252.126:32993/ using document root <none> and password file <none>
I20260928 11:00:44.524248 16414 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:44.524346 16414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:44.524597 16414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 11:00:44 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593244, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:44.591463 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "f66e421f59f24ea69ea8b52ecc9f398f"
format_stamp: "Formatted at 2026-09-28 11:00:44 on dist-test-slave-djjw"
server_key: "5b5c6a412fc0e57b8fa0914871b29954"
server_key_iv: "03d938763e622d0961ece3c7be7b7a27"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:44.592314 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "f66e421f59f24ea69ea8b52ecc9f398f"
format_stamp: "Formatted at 2026-09-28 11:00:44 on dist-test-slave-djjw"
server_key: "5b5c6a412fc0e57b8fa0914871b29954"
server_key_iv: "03d938763e622d0961ece3c7be7b7a27"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:44.597599 16414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 11:00:44.617087 16429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:44.618564 16414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:00:44.618726 16414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "f66e421f59f24ea69ea8b52ecc9f398f"
format_stamp: "Formatted at 2026-09-28 11:00:44 on dist-test-slave-djjw"
server_key: "5b5c6a412fc0e57b8fa0914871b29954"
server_key_iv: "03d938763e622d0961ece3c7be7b7a27"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:44.618935 16414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:44.639575 16414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:44.643560 16414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:44.643847 16414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:44.661873 16414 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:42787
I20260928 11:00:44.661882 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:42787 every 8 connection(s)
I20260928 11:00:44.663230 16414 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:00:44.667896 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:00:44.672617 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 16414
I20260928 11:00:44.672808 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:00:44.673420 4081 external_mini_cluster.cc:1428] Setting key 7176406b05eacf51a58abb625b98b37e
I20260928 11:00:44.679116 16481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Bootstrap starting.
I20260928 11:00:44.682106 16481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:44.683080 16481 log.cc:824] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:44.685814 16481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: No bootstrap required, opened a new log
Sep 28 11:00:44 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:44.694016 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:44.675402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.3.252.126:42787)
0928 11:00:44.676163 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:44.676175 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:44.676217 (+ 42us) server_negotiation.cc:407] Connection header received
0928 11:00:44.677960 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:44.677983 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:44.678367 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:44.678706 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:44.679999 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:44.681283 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:44.682608 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:44.683284 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:44.683451 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:44.687208 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:44.687216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:44.687232 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:44.687268 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:44.690168 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:44.690822 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:44.690829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:44.690836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:44.690979 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:44.691334 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:44.691338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:44.691340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:44.691870 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:44.692190 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:44.692614 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 11:00:44.693003 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":336,"threads_started":1}
I20260928 11:00:44.693779 16481 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:44.694216 16481 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:44.694314 16481 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66e421f59f24ea69ea8b52ecc9f398f, State: Initialized, Role: FOLLOWER
I20260928 11:00:44.694887 16481 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:44.695115 16481 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:00:44.695235 16481 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:00:44.695456 16481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:44.698256 16481 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:44.698698 16481 leader_election.cc:302] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66e421f59f24ea69ea8b52ecc9f398f; no voters:
I20260928 11:00:44.699221 16481 leader_election.cc:288] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:00:44.699384 16486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:00:44.700014 16486 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 LEADER]: Becoming Leader. State: Replica: f66e421f59f24ea69ea8b52ecc9f398f, State: Running, Role: LEADER
I20260928 11:00:44.700577 16486 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:44.701211 16481 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:00:44.703109 16487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } } }
I20260928 11:00:44.703111 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66e421f59f24ea69ea8b52ecc9f398f. Latest consensus state: current_term: 1 leader_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } } }
I20260928 11:00:44.703553 16487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:44.703658 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:44.705856 16491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:00:44.711439 16491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:00:44.713388 16481 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 11:00:44.728363 16491 catalog_manager.cc:1347] Generated new cluster ID: 2a5b81f60f50406ca22ab9ff9eb63c2a
I20260928 11:00:44.728556 16491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:00:44.750068 16491 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:00:44.751441 16491 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:00:44.765161 16491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Generated new TSK 0
I20260928 11:00:44.766438 16491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 11:00:45.520722 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:42787
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 11:00:45 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790593245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 11:00:45.927059 16526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:45.927511 16526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:45.927724 16526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:45.938941 16526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:45.939080 16526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:45.939311 16526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:00:45.954612 16526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.252.84:41655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:45.956494 16526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:45.958262 16526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:45.975548 16536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:45.975921 16537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:45.984050 16539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:45.985913 16526 server_base.cc:1034] running on GCE node
I20260928 11:00:45.986789 16526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:45.988144 16526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:45.989482 16526 hybrid_clock.cc:648] HybridClock initialized: now 1790593245989376 us; error 90 us; skew 500 ppm
I20260928 11:00:45.989907 16526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:45 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:45.994112 16526 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:00:45.996004 16526 webserver.cc:460] Webserver started at http://127.3.252.65:33127/ using document root <none> and password file <none>
I20260928 11:00:45.996892 16526 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:45.997021 16526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:45.997359 16526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 11:00:45 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:46.018095 16526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "e2b7bc19e08d4c12b25f73ab0e320844"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "6973a4fc3b7e2b036c0fa5f39f2014bd"
server_key_iv: "8b0fcdbc948f4ed2f516bd907d36a94b"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:46.019315 16526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "e2b7bc19e08d4c12b25f73ab0e320844"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "6973a4fc3b7e2b036c0fa5f39f2014bd"
server_key_iv: "8b0fcdbc948f4ed2f516bd907d36a94b"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:46.039301 16526 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.006s sys 0.011s
I20260928 11:00:46.064172 16546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:46.065936 16526 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260928 11:00:46.066120 16526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "e2b7bc19e08d4c12b25f73ab0e320844"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "6973a4fc3b7e2b036c0fa5f39f2014bd"
server_key_iv: "8b0fcdbc948f4ed2f516bd907d36a94b"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:46.066339 16526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:46.102777 16526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:46.108266 16526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:46.108676 16526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:46.122390 16526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:00:46.142827 16526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:00:46.143036 16526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 11:00:46.143237 16526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:00:46.143371 16526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 11:00:46 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:46.337291 16526 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:38167
I20260928 11:00:46.337296 16658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:38167 every 8 connection(s)
I20260928 11:00:46.339133 16526 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:00:46.342149 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 16526
I20260928 11:00:46.342358 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:00:46.343181 4081 external_mini_cluster.cc:1428] Setting key 43598ed61154012946258fd9b50a3e97
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Sep 28 11:00:46 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:46.444635 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:46.399043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:57985 (local address 127.3.252.126:42787)
0928 11:00:46.413491 (+ 14448us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:46.413499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:46.414968 (+ 1469us) server_negotiation.cc:407] Connection header received
0928 11:00:46.425803 (+ 10835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:46.425810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:46.425951 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:46.426037 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:46.428340 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:46.429506 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:46.431978 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:46.432617 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:46.432716 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:46.437245 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:46.437252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:46.437257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:46.437297 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:46.440782 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:46.441733 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:46.441751 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:46.441755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:46.441932 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:46.442460 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:46.442483 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:46.442487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:46.442844 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:46.442958 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:46.444013 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0928 11:00:46.444246 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14263,"thread_start_us":14445,"threads_started":1}
I20260928 11:00:46.447212 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:46.398317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:42787 (local address 127.3.252.65:57985)
0928 11:00:46.414314 (+ 15997us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:46.414391 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:46.425393 (+ 11002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:46.426650 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:46.426663 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:46.427293 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:46.428119 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:46.428143 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:46.429724 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:46.429729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:46.431802 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:46.431818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:46.433088 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:46.433095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:46.433401 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:46.434181 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:46.434218 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:46.436970 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:46.440991 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:46.441001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:46.441022 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:46.441595 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:46.442078 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:46.442083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:46.442087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:46.442338 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:46.443084 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:46.443093 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:46.443581 (+ 488us) client_negotiation.cc:769] Sending connection context
0928 11:00:46.445171 (+ 1590us) client_negotiation.cc:240] Negotiation successful
0928 11:00:46.445765 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15622,"thread_start_us":15364,"threads_started":1}
I20260928 11:00:46.449564 16659 heartbeater.cc:347] Connected to a master server at 127.3.252.126:42787
I20260928 11:00:46.449937 16659 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:46.450924 16659 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
W20260928 11:00:46.453073 4081 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.252.66: real 0.107s user 0.004s sys 0.013s
I20260928 11:00:46.453513 16446 ts_manager.cc:194] Registered new tserver with Master: e2b7bc19e08d4c12b25f73ab0e320844 (127.3.252.65:38167)
I20260928 11:00:46.453992 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:42787
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:00:46.455530 16446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:57985
Sep 28 11:00:46 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
W20260928 11:00:46.900698 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:46.901180 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:46.901386 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:46.912676 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:46.912817 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:46.913050 16665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:00:46.928465 16665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.252.84:41655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:46.930179 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:46.931882 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:46.960003 16674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:46.960470 16675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:46.963119 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:46.963526 16665 server_base.cc:1034] running on GCE node
I20260928 11:00:46.964287 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:46.965519 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:46.966760 16665 hybrid_clock.cc:648] HybridClock initialized: now 1790593246966693 us; error 45 us; skew 500 ppm
I20260928 11:00:46.967121 16665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:46 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:46.971168 16665 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:00:46.972944 16665 webserver.cc:460] Webserver started at http://127.3.252.66:34655/ using document root <none> and password file <none>
I20260928 11:00:46.973875 16665 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:46.974009 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:46.974352 16665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 11:00:46 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:46.991694 16665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "4b8165b4911e4ead9cb05f0c350ecfef"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "c3950c1069512b26db7e9cfc8e076e13"
server_key_iv: "f771c8a6c3180ebf4b17adba2808fab5"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:46.992515 16665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "4b8165b4911e4ead9cb05f0c350ecfef"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "c3950c1069512b26db7e9cfc8e076e13"
server_key_iv: "f771c8a6c3180ebf4b17adba2808fab5"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:46.997928 16665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 11:00:47.015404 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:47.017373 16665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 11:00:47.017644 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "4b8165b4911e4ead9cb05f0c350ecfef"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "c3950c1069512b26db7e9cfc8e076e13"
server_key_iv: "f771c8a6c3180ebf4b17adba2808fab5"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:47.018056 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:47.062068 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:47.067682 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:47.068064 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:47.069420 16665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:00:47.071466 16665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:00:47.071606 16665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:47.071766 16665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:00:47.071861 16665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:47.106483 16665 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:33781
I20260928 11:00:47.106491 16801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:33781 every 8 connection(s)
I20260928 11:00:47.108297 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:00:47.118310 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 16665
I20260928 11:00:47.118489 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:00:47.119032 4081 external_mini_cluster.cc:1428] Setting key e9bf263a437b010cf154b6d6a42d4439
Sep 28 11:00:47 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
I20260928 11:00:47.147372 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:47.112329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:55111 (local address 127.3.252.126:42787)
0928 11:00:47.120385 (+ 8056us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:47.120391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:47.121703 (+ 1312us) server_negotiation.cc:407] Connection header received
0928 11:00:47.125248 (+ 3545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:47.125253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:47.125342 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:47.125439 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:47.127603 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.128574 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.130856 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.131404 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.131485 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:47.140132 (+ 8647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:47.140139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:47.140144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:47.140188 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:47.143525 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.144444 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.144450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.144454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.144603 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.145231 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.145236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.145238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.145557 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:47.145680 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:47.146732 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0928 11:00:47.146987 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7918,"thread_start_us":7794,"threads_started":1}
I20260928 11:00:47.149526 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:47.111767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:42787 (local address 127.3.252.66:55111)
0928 11:00:47.121007 (+ 9240us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:47.121066 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:47.124723 (+ 3657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:47.125958 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:47.125973 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:47.126590 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:47.127405 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:47.127429 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.128770 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.128775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:47.130687 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:47.130703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.131748 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.131753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:47.131962 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:47.136928 (+ 4966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:47.136959 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:47.139775 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:47.143722 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:47.143736 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:47.143760 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:47.144297 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:47.144787 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:47.144792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:47.144796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:47.145094 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:47.145683 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:47.145691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:47.146281 (+ 590us) client_negotiation.cc:769] Sending connection context
0928 11:00:47.147592 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0928 11:00:47.148190 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8916,"thread_start_us":8675,"threads_started":1}
I20260928 11:00:47.151214 16802 heartbeater.cc:347] Connected to a master server at 127.3.252.126:42787
I20260928 11:00:47.151540 16802 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:47.152422 16802 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
I20260928 11:00:47.154080 16446 ts_manager.cc:194] Registered new tserver with Master: 4b8165b4911e4ead9cb05f0c350ecfef (127.3.252.66:33781)
I20260928 11:00:47.155169 16446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:55111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 11:00:47.173404 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:42787
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:00:47.459153 16659 heartbeater.cc:503] Master 127.3.252.126:42787 was elected leader, sending a full tablet report...
W20260928 11:00:47.493436 16808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:47.493736 16808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:47.493889 16808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:47.501525 16808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:47.501631 16808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:47.501746 16808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:00:47.511482 16808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.252.84:41655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:47.512806 16808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:47.514078 16808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:47.526710 16816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:47.526891 16813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:47.526728 16814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:47.527482 16808 server_base.cc:1034] running on GCE node
I20260928 11:00:47.528012 16808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:47.528923 16808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:47.530103 16808 hybrid_clock.cc:648] HybridClock initialized: now 1790593247530063 us; error 34 us; skew 500 ppm
I20260928 11:00:47.530376 16808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:47 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:47.533201 16808 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:00:47.534613 16808 webserver.cc:460] Webserver started at http://127.3.252.67:41905/ using document root <none> and password file <none>
I20260928 11:00:47.535236 16808 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:47.535329 16808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:47.535583 16808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 11:00:47 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:47.552582 16808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "e035fbb41fae420c82df470ba8148d44"
format_stamp: "Formatted at 2026-09-28 11:00:47 on dist-test-slave-djjw"
server_key: "1d2ef2bbff915722fdc2a59290cde668"
server_key_iv: "c618ab2f40e811e872ee9f7190806e84"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:47.553650 16808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "e035fbb41fae420c82df470ba8148d44"
format_stamp: "Formatted at 2026-09-28 11:00:47 on dist-test-slave-djjw"
server_key: "1d2ef2bbff915722fdc2a59290cde668"
server_key_iv: "c618ab2f40e811e872ee9f7190806e84"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:47.560043 16808 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 11:00:47.577847 16823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:47.579638 16808 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 11:00:47.579845 16808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "e035fbb41fae420c82df470ba8148d44"
format_stamp: "Formatted at 2026-09-28 11:00:47 on dist-test-slave-djjw"
server_key: "1d2ef2bbff915722fdc2a59290cde668"
server_key_iv: "c618ab2f40e811e872ee9f7190806e84"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:47.580123 16808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:47.604568 16808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:47.609614 16808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:47.610004 16808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:47.611223 16808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:00:47.613075 16808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:00:47.613161 16808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:47.613284 16808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:00:47.613338 16808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:47.643430 16808 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:35595
I20260928 11:00:47.643404 16935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:35595 every 8 connection(s)
I20260928 11:00:47.644960 16808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:00:47.646168 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 16808
I20260928 11:00:47.646327 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:00:47.646912 4081 external_mini_cluster.cc:1428] Setting key 3704d891d5bb7d08d7e88fb8bae7cc42
Sep 28 11:00:47 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:47.665189 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:47.648100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:53967 (local address 127.3.252.126:42787)
0928 11:00:47.648593 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:47.648599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:47.649596 (+ 997us) server_negotiation.cc:407] Connection header received
0928 11:00:47.651544 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:47.651549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:47.651640 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:47.651729 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:47.653079 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.653778 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.655319 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.655727 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.655841 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:47.659143 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:47.659147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:47.659151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:47.659183 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:47.662004 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.662952 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.662957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.662960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.663109 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.663528 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.663533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.663537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.663824 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:47.663970 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:47.664625 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 11:00:47.664838 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260928 11:00:47.666375 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:47.648708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:42787 (local address 127.3.252.67:53967)
0928 11:00:47.649440 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:47.649486 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:47.651293 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:47.651971 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:47.651979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:47.652372 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:47.652927 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:47.652943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.653941 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.653945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:47.655187 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:47.655198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.655936 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.655943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:47.656159 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:47.656773 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:47.656795 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:47.658939 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:47.662169 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:47.662179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:47.662197 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:47.662808 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:47.663224 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:47.663228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:47.663230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:47.663419 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:47.663953 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:47.663960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:47.664287 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 11:00:47.664530 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 11:00:47.665049 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":295,"threads_started":1}
I20260928 11:00:47.667619 16936 heartbeater.cc:347] Connected to a master server at 127.3.252.126:42787
I20260928 11:00:47.667908 16936 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:47.668599 16936 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
I20260928 11:00:47.670063 16446 ts_manager.cc:194] Registered new tserver with Master: e035fbb41fae420c82df470ba8148d44 (127.3.252.67:35595)
I20260928 11:00:47.670962 16446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:53967
I20260928 11:00:47.675519 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:00:47.678005 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 16526
Sep 28 11:00:47 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 11:00:47.701756 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:47.679106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.3.252.65:38167)
0928 11:00:47.685256 (+ 6150us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:47.685263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:47.685291 (+ 28us) server_negotiation.cc:407] Connection header received
0928 11:00:47.685366 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:47.685375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:47.685598 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:47.685730 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:47.687618 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.688673 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.690725 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.691268 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.691370 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:47.696382 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:47.696403 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:47.696420 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:47.696457 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:47.699117 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.699820 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.699827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.699829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.699987 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.700326 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.700329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.700331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.700647 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:47.700893 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:47.701174 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 11:00:47.701365 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5974,"thread_start_us":5811,"threads_started":1}
I20260928 11:00:47.703076 16654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:47.784466 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 16526
I20260928 11:00:47.797633 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 16665
Sep 28 11:00:47 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 11:00:47.812904 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:47.798512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.3.252.66:33781)
0928 11:00:47.799046 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:47.799053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:47.799078 (+ 25us) server_negotiation.cc:407] Connection header received
0928 11:00:47.799136 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:47.799143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:47.799344 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:47.799502 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:47.800680 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.801705 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.803171 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.803699 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.803816 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:47.807841 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:47.807855 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:47.807869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:47.807900 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:47.810303 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.810946 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.810953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.810955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.811080 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.811434 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.811438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.811441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.811756 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:47.812034 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:47.812335 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 11:00:47.812596 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
I20260928 11:00:47.814090 16797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:47.893731 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 16665
I20260928 11:00:47.906395 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 16808
Sep 28 11:00:47 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 11:00:47.922196 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:47.907325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.3.252.67:35595)
0928 11:00:47.907907 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:47.907914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:47.907941 (+ 27us) server_negotiation.cc:407] Connection header received
0928 11:00:47.907996 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:47.908002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:47.908210 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:47.908329 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:47.909627 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.910712 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.912090 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:47.912653 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:47.912754 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:47.916884 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:47.916897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:47.916913 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:47.916949 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:47.919565 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.920232 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.920239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.920241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.920371 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:47.920701 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:47.920706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:47.920708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:47.921086 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:47.921329 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:47.921569 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 11:00:47.921778 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
I20260928 11:00:47.923465 16931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:48.003800 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 16808
I20260928 11:00:48.016530 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 16414
I20260928 11:00:48.017704 16476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:48.122411 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 16414
I20260928 11:00:48.141422 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:42787
--webserver_interface=127.3.252.126
--webserver_port=32993
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:42787
--ranger_config_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:00:48.454305 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:48.454586 16948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:48.454710 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:48.461783 16948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:00:48.461910 16948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:48.461989 16948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:48.462049 16948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:00:48.462103 16948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:00:48.472370 16948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:42787
--ranger_config_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=32993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:48.473647 16948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:48.474967 16948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:48.486543 16953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:48.486554 16954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:48.486549 16956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:48.487368 16948 server_base.cc:1034] running on GCE node
I20260928 11:00:48.488109 16948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:48.489506 16948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:48.490702 16948 hybrid_clock.cc:648] HybridClock initialized: now 1790593248490647 us; error 48 us; skew 500 ppm
I20260928 11:00:48.491055 16948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:48 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:48.494202 16948 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:00:48.495645 16948 webserver.cc:460] Webserver started at http://127.3.252.126:32993/ using document root <none> and password file <none>
I20260928 11:00:48.496272 16948 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:48.496374 16948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 11:00:48 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:48.545286 16948 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:00:48.548391 16964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:48.549734 16948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 11:00:48.549923 16948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "f66e421f59f24ea69ea8b52ecc9f398f"
format_stamp: "Formatted at 2026-09-28 11:00:44 on dist-test-slave-djjw"
server_key: "5b5c6a412fc0e57b8fa0914871b29954"
server_key_iv: "03d938763e622d0961ece3c7be7b7a27"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:48.550837 16948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:48.561715 16948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:48.565534 16948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:48.565785 16948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:48.582674 16948 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:42787
I20260928 11:00:48.582691 17015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:42787 every 8 connection(s)
I20260928 11:00:48.584079 16948 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:00:48.590771 17016 sys_catalog.cc:263] Verifying existing consensus state
I20260928 11:00:48.592590 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 16948
I20260928 11:00:48.593369 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:38167
--local_ip_for_outbound_sockets=127.3.252.65
--tserver_master_addrs=127.3.252.126:42787
--webserver_port=33127
--webserver_interface=127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:00:48.594163 17016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Bootstrap starting.
I20260928 11:00:48.612219 17016 log.cc:824] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:48.621281 17016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 11:00:48.621806 17016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Bootstrap complete.
I20260928 11:00:48.630384 17016 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:48.631240 17016 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66e421f59f24ea69ea8b52ecc9f398f, State: Initialized, Role: FOLLOWER
I20260928 11:00:48.631816 17016 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:48.632032 17016 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:00:48.632143 17016 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:00:48.632293 17016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 1 FOLLOWER]: Advancing to term 2
I20260928 11:00:48.635990 17016 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:48.636447 17016 leader_election.cc:302] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66e421f59f24ea69ea8b52ecc9f398f; no voters:
I20260928 11:00:48.637030 17016 leader_election.cc:288] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 11:00:48.637247 17020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 11:00:48.638253 17020 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [term 2 LEADER]: Becoming Leader. State: Replica: f66e421f59f24ea69ea8b52ecc9f398f, State: Running, Role: LEADER
I20260928 11:00:48.639022 17016 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:00:48.638962 17020 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } }
I20260928 11:00:48.643608 17021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } } }
I20260928 11:00:48.644037 17021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:48.644711 17022 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66e421f59f24ea69ea8b52ecc9f398f. Latest consensus state: current_term: 2 leader_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66e421f59f24ea69ea8b52ecc9f398f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 42787 } } }
I20260928 11:00:48.645093 17022 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:48.649256 17030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:00:48.654682 17030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:00:48.655647 17016 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 11:00:48.656121 17030 catalog_manager.cc:1259] Loaded cluster ID: 2a5b81f60f50406ca22ab9ff9eb63c2a
I20260928 11:00:48.656279 17030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:00:48.660184 17030 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:00:48.661681 17030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Loaded TSK: 0
I20260928 11:00:48.663281 17030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 11:00:48.982621 17018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:48.983137 17018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:48.983372 17018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:48.995154 17018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:48.995311 17018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:48.995594 17018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:00:49.011446 17018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:38167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=33127
--tserver_master_addrs=127.3.252.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:49.013236 17018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:49.015053 17018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:49.048444 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:49.049250 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:49.049499 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:49.055644 17018 server_base.cc:1034] running on GCE node
I20260928 11:00:49.056552 17018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:49.057996 17018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:49.059309 17018 hybrid_clock.cc:648] HybridClock initialized: now 1790593249059235 us; error 60 us; skew 500 ppm
I20260928 11:00:49.059749 17018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:49 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:49.064258 17018 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:00:49.082247 17018 webserver.cc:460] Webserver started at http://127.3.252.65:33127/ using document root <none> and password file <none>
I20260928 11:00:49.083444 17018 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:49.083701 17018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 11:00:49 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:49.120788 17018 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260928 11:00:49.131057 17064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:49.132990 17018 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260928 11:00:49.133244 17018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "e2b7bc19e08d4c12b25f73ab0e320844"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "6973a4fc3b7e2b036c0fa5f39f2014bd"
server_key_iv: "8b0fcdbc948f4ed2f516bd907d36a94b"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:49.134821 17018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:49.206792 17018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:49.210722 17018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:49.210999 17018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:49.212045 17018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:00:49.213435 17018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:00:49.213534 17018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:49.213636 17018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:00:49.213708 17018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:49.278901 17018 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:38167
I20260928 11:00:49.278942 17177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:38167 every 8 connection(s)
I20260928 11:00:49.280352 17018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:00:49.291090 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17018
I20260928 11:00:49.291909 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:33781
--local_ip_for_outbound_sockets=127.3.252.66
--tserver_master_addrs=127.3.252.126:42787
--webserver_port=34655
--webserver_interface=127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 11:00:49 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:49.316737 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:49.283681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:60903 (local address 127.3.252.126:42787)
0928 11:00:49.284454 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:49.284475 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:49.286585 (+ 2110us) server_negotiation.cc:407] Connection header received
0928 11:00:49.289099 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:49.289121 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:49.289634 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:49.290044 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:49.298148 (+ 8104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.300169 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.302808 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.303608 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.303727 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:49.307714 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:49.307723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:49.307739 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:49.307786 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:49.310980 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:49.312014 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:49.312024 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:49.312032 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:49.312237 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:49.312798 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:49.312804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:49.312807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:49.313512 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:49.313666 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:49.314778 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0928 11:00:49.315271 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":547,"threads_started":1}
I20260928 11:00:49.319487 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:49.283561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:42787 (local address 127.3.252.65:60903)
0928 11:00:49.286405 (+ 2844us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:49.286458 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:49.288387 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:49.296215 (+ 7828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:49.296229 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:49.296881 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:49.297858 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:49.297892 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.300549 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.300556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:49.302598 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:49.302625 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.304071 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.304076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:49.304260 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:49.304929 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:49.304961 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:49.307434 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:49.311178 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:49.311189 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:49.311208 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:49.311836 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:49.312386 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:49.312390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:49.312407 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:49.312668 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:49.313752 (+ 1084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:49.313759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:49.314183 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 11:00:49.317633 (+ 3450us) client_negotiation.cc:240] Negotiation successful
0928 11:00:49.318241 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2552,"thread_start_us":2379,"threads_started":1}
I20260928 11:00:49.323539 17178 heartbeater.cc:347] Connected to a master server at 127.3.252.126:42787
I20260928 11:00:49.323871 17178 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:49.324796 17178 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
W20260928 11:00:49.327662 17178 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:42787 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 11:00:49.327870 17178 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:49.328145 17178 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
I20260928 11:00:49.329183 17178 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:49.329495 17178 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
W20260928 11:00:49.330490 17178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 11:00:49.710110 17182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:49.710403 17182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:49.710546 17182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:49.717904 17182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:49.718022 17182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:49.718171 17182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:00:49.729058 17182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:33781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=34655
--tserver_master_addrs=127.3.252.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:49.730710 17182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:49.732362 17182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:49.747287 17197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:49.748091 17196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:49.751238 17199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:49.751503 17182 server_base.cc:1034] running on GCE node
I20260928 11:00:49.752344 17182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:49.753609 17182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:49.754861 17182 hybrid_clock.cc:648] HybridClock initialized: now 1790593249754776 us; error 57 us; skew 500 ppm
I20260928 11:00:49.755241 17182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:49 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:49.764113 17182 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:00:49.766271 17182 webserver.cc:460] Webserver started at http://127.3.252.66:34655/ using document root <none> and password file <none>
I20260928 11:00:49.767266 17182 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:49.767432 17182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 11:00:49 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:49.791262 17182 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 11:00:49.794776 17207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:49.796499 17182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:00:49.796671 17182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "4b8165b4911e4ead9cb05f0c350ecfef"
format_stamp: "Formatted at 2026-09-28 11:00:46 on dist-test-slave-djjw"
server_key: "c3950c1069512b26db7e9cfc8e076e13"
server_key_iv: "f771c8a6c3180ebf4b17adba2808fab5"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:49.798051 17182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:49.874272 17182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:49.878186 17182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:49.878485 17182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:49.879522 17182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:00:49.880926 17182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:00:49.881023 17182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:49.881129 17182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:00:49.881192 17182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 11:00:49 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:49.962695 17182 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:33781
I20260928 11:00:49.962764 17321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:33781 every 8 connection(s)
I20260928 11:00:49.964514 17182 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:00:49.968869 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17182
I20260928 11:00:49.969620 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.252.84:41655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:35595
--local_ip_for_outbound_sockets=127.3.252.67
--tserver_master_addrs=127.3.252.126:42787
--webserver_port=41905
--webserver_interface=127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 11:00:49 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:49.989059 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:49.967386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:54801 (local address 127.3.252.126:42787)
0928 11:00:49.968428 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:49.968434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:49.971523 (+ 3089us) server_negotiation.cc:407] Connection header received
0928 11:00:49.971595 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:49.971599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:49.971681 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:49.971772 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:49.973746 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.974845 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.976953 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.977515 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.977607 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:49.981939 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:49.981945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:49.981951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:49.981992 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:49.985230 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:49.986251 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:49.986257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:49.986261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:49.986423 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:49.986980 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:49.986985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:49.986989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:49.987386 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:49.987540 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:49.988460 (+ 920us) server_negotiation.cc:299] Negotiation successful
0928 11:00:49.988701 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":738,"threads_started":1}
I20260928 11:00:49.990389 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:49.967983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:42787 (local address 127.3.252.66:54801)
0928 11:00:49.968863 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:49.968920 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:49.971181 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:49.972183 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:49.972195 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:49.972757 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:49.973549 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:49.973574 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.975063 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.975068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:49.976791 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:49.976805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:49.977916 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:49.977923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:49.978212 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:49.979069 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:49.979101 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:49.981632 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:49.985423 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:49.985433 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:49.985452 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:49.986086 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:49.986540 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:49.986544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:49.986552 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:49.986828 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:49.987531 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:49.987539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:49.987996 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 11:00:49.988300 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 11:00:49.988867 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":363,"threads_started":1}
I20260928 11:00:49.992429 17322 heartbeater.cc:347] Connected to a master server at 127.3.252.126:42787
I20260928 11:00:49.992779 17322 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:49.993660 17322 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
I20260928 11:00:49.995854 16981 ts_manager.cc:194] Registered new tserver with Master: 4b8165b4911e4ead9cb05f0c350ecfef (127.3.252.66:33781)
I20260928 11:00:49.998128 16981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:54801
I20260928 11:00:50.330664 17178 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:50.331144 17178 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
I20260928 11:00:50.333012 16981 ts_manager.cc:194] Registered new tserver with Master: e2b7bc19e08d4c12b25f73ab0e320844 (127.3.252.65:38167)
I20260928 11:00:50.335002 16981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:60903
Sep 28 11:00:50 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
W20260928 11:00:50.375216 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:50.375742 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:50.375946 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:50.387337 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:50.387565 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:50.387775 17326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:00:50.400607 17326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.252.84:41655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:35595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=41905
--tserver_master_addrs=127.3.252.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:50.402701 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:50.404580 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:50.442413 17332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:50.454849 17333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:50.457919 17326 server_base.cc:1034] running on GCE node
W20260928 11:00:50.459782 17335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:50.460685 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:50.471275 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:50.474769 17326 hybrid_clock.cc:648] HybridClock initialized: now 1790593250474668 us; error 90 us; skew 500 ppm
I20260928 11:00:50.475306 17326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:50 dist-test-slave-djjw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:50.479343 17326 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:00:50.489348 17326 webserver.cc:460] Webserver started at http://127.3.252.67:41905/ using document root <none> and password file <none>
I20260928 11:00:50.490406 17326 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:50.490617 17326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 11:00:50 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:50.519348 17326 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 11:00:50.522327 17343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:50.523675 17326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:00:50.523830 17326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "e035fbb41fae420c82df470ba8148d44"
format_stamp: "Formatted at 2026-09-28 11:00:47 on dist-test-slave-djjw"
server_key: "1d2ef2bbff915722fdc2a59290cde668"
server_key_iv: "c618ab2f40e811e872ee9f7190806e84"
server_key_version: "kuduclusterkey@0"
I20260928 11:00:50.524807 17326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:50.549120 17326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:50.554685 17326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:50.555052 17326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:50.566322 17326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:00:50.592012 17326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:00:50.592216 17326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 11:00:50.592365 17326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:00:50.592468 17326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 11:00:50 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for HTTP/127.3.252.84@KRBTEST.COM
I20260928 11:00:50.938124 17326 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:35595
I20260928 11:00:50.938179 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:35595 every 8 connection(s)
I20260928 11:00:50.940115 17326 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:00:50.943331 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17326
I20260928 11:00:51.002871 17322 heartbeater.cc:503] Master 127.3.252.126:42787 was elected leader, sending a full tablet report...
I20260928 11:00:51.005931 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:50.944381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.3.252.126:42787)
0928 11:00:50.964642 (+ 20261us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:50.964649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:50.964673 (+ 24us) server_negotiation.cc:407] Connection header received
0928 11:00:50.964745 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:50.964751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:50.964846 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:50.964970 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:50.967257 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:50.968516 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:50.979251 (+ 10735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:50.979994 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:50.980144 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:50.997390 (+ 17246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:50.997399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:50.997404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:50.997466 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:51.001112 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.003349 (+ 2237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.003355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.003358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.003523 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.004009 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.004013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.004016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.004384 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:51.004625 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.005227 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.005491 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20104,"thread_start_us":20694,"threads_started":1}
Sep 28 11:00:51 dist-test-slave-djjw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:51.017056 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:50.969169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:60163 (local address 127.3.252.126:42787)
0928 11:00:50.987143 (+ 17974us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:50.987150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:50.991316 (+ 4166us) server_negotiation.cc:407] Connection header received
0928 11:00:50.991395 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:50.991400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:50.991493 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:50.991596 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:51.000753 (+ 9157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.002093 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.004614 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.005308 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.005416 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.010407 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:51.010414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:51.010419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:51.010459 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:51.013431 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.014440 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.014446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.014449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.014621 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.015183 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.015188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.015192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.015549 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:51.015694 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.016426 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.016651 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17798,"thread_start_us":17978,"threads_started":1}
I20260928 11:00:51.018015 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:50.968447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:42787 (local address 127.3.252.67:60163)
0928 11:00:50.987932 (+ 19485us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:50.987997 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:50.990812 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:50.998987 (+ 8175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:50.999002 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:50.999630 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:51.000506 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.000535 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.002373 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.002379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.004420 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.004436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.005735 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.005742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.006062 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.006749 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:51.006779 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:51.010096 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:51.013577 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:51.013587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:51.013606 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:51.014283 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:51.014761 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:51.014766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:51.014770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:51.015043 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:51.015694 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:51.015700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:51.016041 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 11:00:51.016323 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 11:00:51.016763 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19083,"thread_start_us":25326,"threads_started":1}
I20260928 11:00:51.019520 17457 heartbeater.cc:347] Connected to a master server at 127.3.252.126:42787
I20260928 11:00:51.019869 17457 heartbeater.cc:464] Registering TS with master...
I20260928 11:00:51.020732 17457 heartbeater.cc:511] Master 127.3.252.126:42787 requested a full tablet report, sending...
I20260928 11:00:51.022483 16980 ts_manager.cc:194] Registered new tserver with Master: e035fbb41fae420c82df470ba8148d44 (127.3.252.67:35595)
I20260928 11:00:51.023428 16980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:60163
I20260928 11:00:51.025352 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:00:51.042600 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.029163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.3.252.126:42787)
0928 11:00:51.029337 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.029344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.029879 (+ 535us) server_negotiation.cc:407] Connection header received
0928 11:00:51.030058 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.030062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.030139 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.030222 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:51.032250 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.032998 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.034505 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.034913 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.035013 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.037345 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:51.037350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:51.037353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:51.037385 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:51.039993 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.040726 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.040730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.040732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.040851 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.041222 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.041225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.041228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.041510 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:51.041727 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.042082 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.042300 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 11:00:51.047585 16980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:00:51.054822 16980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:00:51.088891 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.068027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:35595 (local address 127.0.0.1:43486)
0928 11:00:51.080161 (+ 12134us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:51.080179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:51.080383 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:51.081138 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:51.081146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:51.081179 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:00:51.081748 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.081765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.083854 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.083860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.086458 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.086478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.087873 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.087880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.088075 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.088135 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 11:00:51.088245 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 11:00:51.088432 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12009,"mutex_wait_us":67,"thread_start_us":11857,"threads_started":1}
I20260928 11:00:51.089407 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.067841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:33781 (local address 127.0.0.1:57878)
0928 11:00:51.079469 (+ 11628us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:51.079531 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:51.080390 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:51.081378 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:51.081382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:51.081401 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:00:51.081890 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.081901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.084365 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.084370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.086548 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.086559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.088567 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.088573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.088800 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.088831 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 11:00:51.088903 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 11:00:51.089070 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11323,"mutex_wait_us":99,"thread_start_us":11174,"threads_started":1}
I20260928 11:00:51.089511 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.068164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.3.252.66:33781)
0928 11:00:51.068926 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.068935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.079698 (+ 10763us) server_negotiation.cc:407] Connection header received
0928 11:00:51.080665 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.080672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.080908 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.081079 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:00:51.082078 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.083719 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.086688 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.088042 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.088142 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.088351 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.088993 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.089171 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":447,"threads_started":1}
I20260928 11:00:51.090032 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.068496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.3.252.67:35595)
0928 11:00:51.077359 (+ 8863us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.077371 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.080282 (+ 2911us) server_negotiation.cc:407] Connection header received
0928 11:00:51.080632 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.080641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.080907 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.081992 (+ 1085us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:00:51.082281 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.083723 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.086653 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.087638 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.089215 (+ 1577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.089447 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.089515 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.089683 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8697,"thread_start_us":8568,"threads_started":1}
I20260928 11:00:51.090080 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.069073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:38167 (local address 127.0.0.1:50928)
0928 11:00:51.080958 (+ 11885us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:51.080977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:51.081081 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:51.081974 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:51.081978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:51.082001 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:00:51.082308 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.082315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.083614 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.083619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.086303 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.086425 (+ 122us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.088534 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.088543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.088727 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.088814 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 11:00:51.088903 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 11:00:51.089700 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11770,"mutex_wait_us":44,"thread_start_us":11658,"threads_started":1}
I20260928 11:00:51.090452 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.069383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.3.252.65:38167)
0928 11:00:51.078979 (+ 9596us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.078988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.081198 (+ 2210us) server_negotiation.cc:407] Connection header received
0928 11:00:51.081303 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.081310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.081522 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.081692 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:00:51.082492 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.083476 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.086548 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.087977 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.088115 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.088351 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.089961 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.090131 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9434,"thread_start_us":9270,"threads_started":1}
I20260928 11:00:51.093250 17257 tablet_service.cc:1467] Processing CreateTablet for tablet 9132fd27e8004a79a7d9cd153872d074 (DEFAULT_TABLE table=test-table [id=cab1a520a85e4aa6a3c5b31fbc91e6e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:00:51.093724 17392 tablet_service.cc:1467] Processing CreateTablet for tablet 9132fd27e8004a79a7d9cd153872d074 (DEFAULT_TABLE table=test-table [id=cab1a520a85e4aa6a3c5b31fbc91e6e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:00:51.093874 17113 tablet_service.cc:1467] Processing CreateTablet for tablet 9132fd27e8004a79a7d9cd153872d074 (DEFAULT_TABLE table=test-table [id=cab1a520a85e4aa6a3c5b31fbc91e6e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:00:51.094621 17257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9132fd27e8004a79a7d9cd153872d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:00:51.095062 17392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9132fd27e8004a79a7d9cd153872d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:00:51.095077 17113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9132fd27e8004a79a7d9cd153872d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:00:51.106343 17485 tablet_bootstrap.cc:492] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Bootstrap starting.
I20260928 11:00:51.110100 17485 tablet_bootstrap.cc:654] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:51.110572 17487 tablet_bootstrap.cc:492] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Bootstrap starting.
I20260928 11:00:51.111661 17485 log.cc:824] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:51.113140 17486 tablet_bootstrap.cc:492] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Bootstrap starting.
I20260928 11:00:51.114102 17487 tablet_bootstrap.cc:654] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:51.115312 17485 tablet_bootstrap.cc:492] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: No bootstrap required, opened a new log
I20260928 11:00:51.115296 17487 log.cc:824] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:51.115609 17485 ts_tablet_manager.cc:1397] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 11:00:51.116937 17486 tablet_bootstrap.cc:654] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:51.117952 17486 log.cc:824] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:51.120833 17487 tablet_bootstrap.cc:492] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: No bootstrap required, opened a new log
I20260928 11:00:51.121001 17486 tablet_bootstrap.cc:492] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: No bootstrap required, opened a new log
I20260928 11:00:51.121099 17487 ts_tablet_manager.cc:1397] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 11:00:51.121273 17486 ts_tablet_manager.cc:1397] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 11:00:51.126703 17485 raft_consensus.cc:348] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.127199 17485 raft_consensus.cc:374] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:51.127347 17485 raft_consensus.cc:729] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8165b4911e4ead9cb05f0c350ecfef, State: Initialized, Role: FOLLOWER
I20260928 11:00:51.128057 17485 consensus_queue.cc:260] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.128985 17487 raft_consensus.cc:348] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.129336 17487 raft_consensus.cc:374] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:51.129431 17487 raft_consensus.cc:729] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e035fbb41fae420c82df470ba8148d44, State: Initialized, Role: FOLLOWER
I20260928 11:00:51.129931 17485 ts_tablet_manager.cc:1428] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 11:00:51.130002 17487 consensus_queue.cc:260] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.131067 17457 heartbeater.cc:503] Master 127.3.252.126:42787 was elected leader, sending a full tablet report...
I20260928 11:00:51.131619 17487 ts_tablet_manager.cc:1428] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260928 11:00:51.132131 17486 raft_consensus.cc:348] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.132524 17486 raft_consensus.cc:374] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:51.132608 17486 raft_consensus.cc:729] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b7bc19e08d4c12b25f73ab0e320844, State: Initialized, Role: FOLLOWER
I20260928 11:00:51.133183 17486 consensus_queue.cc:260] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.134665 17178 heartbeater.cc:503] Master 127.3.252.126:42787 was elected leader, sending a full tablet report...
I20260928 11:00:51.135396 17486 ts_tablet_manager.cc:1428] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 11:00:51.191758 17493 raft_consensus.cc:482] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:00:51.192075 17493 raft_consensus.cc:504] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.194173 17493 leader_election.cc:288] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b8165b4911e4ead9cb05f0c350ecfef (127.3.252.66:33781), e035fbb41fae420c82df470ba8148d44 (127.3.252.67:35595)
I20260928 11:00:51.200098 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.194865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:36657 (local address 127.3.252.66:33781)
0928 11:00:51.195067 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.195079 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.195148 (+ 69us) server_negotiation.cc:407] Connection header received
0928 11:00:51.195359 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.195364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.195428 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.195505 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:00:51.196096 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.196793 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.198519 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.199281 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.199425 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.199510 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.199751 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.199872 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 11:00:51.200263 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.194427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:33781 (local address 127.3.252.65:36657)
0928 11:00:51.195060 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:51.195083 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:51.195232 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:51.195536 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:51.195540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:51.195566 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:00:51.195965 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.195976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.196917 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.196922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.198383 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.198394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.199467 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.199473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.199615 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.199638 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 11:00:51.199718 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 11:00:51.199839 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":3,"thread_start_us":519,"threads_started":1}
I20260928 11:00:51.200785 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.194598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:58461 (local address 127.3.252.67:35595)
0928 11:00:51.194797 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.194803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.196150 (+ 1347us) server_negotiation.cc:407] Connection header received
0928 11:00:51.196285 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.196291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.196352 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.196421 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:00:51.196879 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.197572 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.199233 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.199982 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.200119 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.200187 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.200516 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.200617 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 11:00:51.201305 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.194540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:35595 (local address 127.3.252.65:58461)
0928 11:00:51.196050 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:51.196065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:51.196181 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:51.196444 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:51.196448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:51.196469 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:00:51.196750 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.196758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.197697 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.197702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.199092 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:51.199104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.200182 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.200188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:51.200347 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.200376 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 11:00:51.200936 (+ 560us) client_negotiation.cc:240] Negotiation successful
0928 11:00:51.201055 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"mutex_wait_us":19,"thread_start_us":1197,"threads_started":1}
I20260928 11:00:51.201148 17277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132fd27e8004a79a7d9cd153872d074" candidate_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" is_pre_election: true
I20260928 11:00:51.201637 17277 raft_consensus.cc:2454] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b7bc19e08d4c12b25f73ab0e320844 in term 0.
I20260928 11:00:51.202152 17412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132fd27e8004a79a7d9cd153872d074" candidate_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e035fbb41fae420c82df470ba8148d44" is_pre_election: true
I20260928 11:00:51.202255 17065 leader_election.cc:302] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8165b4911e4ead9cb05f0c350ecfef, e2b7bc19e08d4c12b25f73ab0e320844; no voters:
I20260928 11:00:51.202591 17493 raft_consensus.cc:2781] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:00:51.202602 17412 raft_consensus.cc:2454] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b7bc19e08d4c12b25f73ab0e320844 in term 0.
I20260928 11:00:51.202744 17493 raft_consensus.cc:482] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:00:51.202863 17493 raft_consensus.cc:3037] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:51.205086 17493 raft_consensus.cc:504] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.205628 17493 leader_election.cc:288] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [CANDIDATE]: Term 1 election: Requested vote from peers 4b8165b4911e4ead9cb05f0c350ecfef (127.3.252.66:33781), e035fbb41fae420c82df470ba8148d44 (127.3.252.67:35595)
I20260928 11:00:51.206084 17412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132fd27e8004a79a7d9cd153872d074" candidate_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e035fbb41fae420c82df470ba8148d44"
I20260928 11:00:51.206082 17277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132fd27e8004a79a7d9cd153872d074" candidate_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8165b4911e4ead9cb05f0c350ecfef"
I20260928 11:00:51.206256 17277 raft_consensus.cc:3037] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:51.206249 17412 raft_consensus.cc:3037] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:51.208349 17412 raft_consensus.cc:2454] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b7bc19e08d4c12b25f73ab0e320844 in term 1.
I20260928 11:00:51.208400 17277 raft_consensus.cc:2454] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b7bc19e08d4c12b25f73ab0e320844 in term 1.
I20260928 11:00:51.208767 17065 leader_election.cc:302] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b8165b4911e4ead9cb05f0c350ecfef, e035fbb41fae420c82df470ba8148d44, e2b7bc19e08d4c12b25f73ab0e320844; no voters:
I20260928 11:00:51.209076 17493 raft_consensus.cc:2781] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:00:51.209571 17493 raft_consensus.cc:686] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 1 LEADER]: Becoming Leader. State: Replica: e2b7bc19e08d4c12b25f73ab0e320844, State: Running, Role: LEADER
I20260928 11:00:51.210103 17493 consensus_queue.cc:237] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } }
I20260928 11:00:51.214255 16981 catalog_manager.cc:5495] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2b7bc19e08d4c12b25f73ab0e320844 (127.3.252.65). New cstate: current_term: 1 leader_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b7bc19e08d4c12b25f73ab0e320844" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 38167 } health_report { overall_health: HEALTHY } } }
W20260928 11:00:51.218398 17323 tablet.cc:2369] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:00:51.251083 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.245812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.3.252.65:38167)
0928 11:00:51.246022 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.246028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.246085 (+ 57us) server_negotiation.cc:407] Connection header received
0928 11:00:51.246304 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.246308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.246402 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.246532 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:00:51.246874 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.247588 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.249007 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.249400 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.249507 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.249998 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:00:51.250518 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:00:51.250648 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.250747 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.250828 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 11:00:51.251891 17460 tablet.cc:2369] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:00:51.259409 17277 raft_consensus.cc:1261] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 1 FOLLOWER]: Refusing update from remote peer e2b7bc19e08d4c12b25f73ab0e320844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:00:51.259413 17412 raft_consensus.cc:1261] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 1 FOLLOWER]: Refusing update from remote peer e2b7bc19e08d4c12b25f73ab0e320844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:00:51.260362 17493 consensus_queue.cc:1035] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8165b4911e4ead9cb05f0c350ecfef" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 33781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:00:51.260696 17496 consensus_queue.cc:1035] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e035fbb41fae420c82df470ba8148d44" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:00:51.290275 16980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:00:51.290575 16980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:00:51.294539 16980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f66e421f59f24ea69ea8b52ecc9f398f: Sending DeleteTablet for 3 replicas of tablet 9132fd27e8004a79a7d9cd153872d074
I20260928 11:00:51.295789 17257 tablet_service.cc:1514] Processing DeleteTablet for tablet 9132fd27e8004a79a7d9cd153872d074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:00:51 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:57878
I20260928 11:00:51.296147 17113 tablet_service.cc:1514] Processing DeleteTablet for tablet 9132fd27e8004a79a7d9cd153872d074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:00:51 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:50928
I20260928 11:00:51.296521 17392 tablet_service.cc:1514] Processing DeleteTablet for tablet 9132fd27e8004a79a7d9cd153872d074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:00:51 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:43486
I20260928 11:00:51.296875 17509 tablet_replica.cc:330] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: stopping tablet replica
I20260928 11:00:51.297329 17509 raft_consensus.cc:2229] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:00:51.297878 17509 raft_consensus.cc:2258] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:00:51.298859 17511 tablet_replica.cc:330] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: stopping tablet replica
I20260928 11:00:51.299245 17511 raft_consensus.cc:2229] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:00:51.299671 17511 raft_consensus.cc:2258] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:00:51.299902 17509 ts_tablet_manager.cc:1905] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:00:51.301541 17511 ts_tablet_manager.cc:1905] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:00:51.303400 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17018
I20260928 11:00:51.306692 17509 ts_tablet_manager.cc:1918] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:00:51.306860 17509 log.cc:1196] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/9132fd27e8004a79a7d9cd153872d074
I20260928 11:00:51.307334 17509 ts_tablet_manager.cc:1939] T 9132fd27e8004a79a7d9cd153872d074 P 4b8165b4911e4ead9cb05f0c350ecfef: Deleting consensus metadata
I20260928 11:00:51.308122 17511 ts_tablet_manager.cc:1918] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:00:51.308264 17511 log.cc:1196] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/9132fd27e8004a79a7d9cd153872d074
I20260928 11:00:51.308741 17511 ts_tablet_manager.cc:1939] T 9132fd27e8004a79a7d9cd153872d074 P e035fbb41fae420c82df470ba8148d44: Deleting consensus metadata
I20260928 11:00:51.309182 16965 catalog_manager.cc:4841] TS 4b8165b4911e4ead9cb05f0c350ecfef (127.3.252.66:33781): tablet 9132fd27e8004a79a7d9cd153872d074 (table test-table [id=cab1a520a85e4aa6a3c5b31fbc91e6e8]) successfully deleted
I20260928 11:00:51.310330 16967 catalog_manager.cc:4841] TS e035fbb41fae420c82df470ba8148d44 (127.3.252.67:35595): tablet 9132fd27e8004a79a7d9cd153872d074 (table test-table [id=cab1a520a85e4aa6a3c5b31fbc91e6e8]) successfully deleted
I20260928 11:00:51.311189 17510 tablet_replica.cc:330] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: stopping tablet replica
I20260928 11:00:51.311671 17510 raft_consensus.cc:2229] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:00:51.312368 17510 raft_consensus.cc:2258] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:00:51.313928 17510 ts_tablet_manager.cc:1905] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:00:51.320695 17510 ts_tablet_manager.cc:1918] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:00:51.320891 17510 log.cc:1196] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/9132fd27e8004a79a7d9cd153872d074
I20260928 11:00:51.321432 17510 ts_tablet_manager.cc:1939] T 9132fd27e8004a79a7d9cd153872d074 P e2b7bc19e08d4c12b25f73ab0e320844: Deleting consensus metadata
I20260928 11:00:51.322196 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.304471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.3.252.65:38167)
0928 11:00:51.304642 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.304648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.304732 (+ 84us) server_negotiation.cc:407] Connection header received
0928 11:00:51.304927 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.304932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.305006 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.305109 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:51.307231 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.308337 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.310610 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.311373 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.311493 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.314794 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:51.314808 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:51.314825 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:51.314869 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:51.318973 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.319835 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.319845 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.319849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.320046 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.320467 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.320472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.320475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.320958 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:51.321275 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.321580 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.321808 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 11:00:51.322970 17173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:51.323271 16966 catalog_manager.cc:4841] TS e2b7bc19e08d4c12b25f73ab0e320844 (127.3.252.65:38167): tablet 9132fd27e8004a79a7d9cd153872d074 (table test-table [id=cab1a520a85e4aa6a3c5b31fbc91e6e8]) successfully deleted
I20260928 11:00:51.417691 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17018
I20260928 11:00:51.431895 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17182
I20260928 11:00:51.448210 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.432619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.3.252.66:33781)
0928 11:00:51.432795 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.432800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.432818 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:00:51.432954 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.432958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.433019 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.433107 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:51.434304 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.435010 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.436180 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.436564 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.436649 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.443104 (+ 6455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:51.443119 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:51.443137 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:51.443172 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:51.445983 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.446554 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.446560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.446563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.446690 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.446964 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.446967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.446969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.447289 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:51.447512 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.447702 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.447868 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 11:00:51.448843 17317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:51.535655 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17182
I20260928 11:00:51.549474 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17326
I20260928 11:00:51.562152 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:51.550195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.3.252.67:35595)
0928 11:00:51.550363 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:51.550369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:51.550389 (+ 20us) server_negotiation.cc:407] Connection header received
0928 11:00:51.550591 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:51.550597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:51.550674 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:51.550755 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:51.551916 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.552619 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.553874 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:51.554337 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:51.554447 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:51.556846 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:51.556859 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:51.556874 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:51.556913 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:51.559364 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.559995 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.560004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.560006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.560119 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:51.560465 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:51.560469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:51.560472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:51.560817 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:51.561093 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:51.561379 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 11:00:51.561583 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 11:00:51.562772 17452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:51.649894 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17326
I20260928 11:00:51.663352 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 16948
I20260928 11:00:51.664515 17011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:51.775695 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 16948
I20260928 11:00:51.793653 4081 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 11:00:52.129060 4081 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 11:00:52.129312 4081 mini_ranger.cc:66] Stopping Ranger...
I20260928 11:00:52.464370 4081 mini_ranger.cc:68] Stopped Ranger
2026-09-28 11:00:52.464 UTC [16000] LOG: received smart shutdown request
2026-09-28 11:00:52.467 UTC [16000] LOG: background worker "logical replication launcher" (PID 16009) exited with exit code 1
2026-09-28 11:00:52.468 UTC [16004] LOG: shutting down
2026-09-28 11:00:52.626 UTC [16000] LOG: database system is shut down
2026-09-28T11:00:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51435 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:00:52 dist-test-slave-djjw krb5kdc[17524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:00:52 dist-test-slave-djjw krb5kdc[17524](info): set up 2 sockets
Sep 28 11:00:52 dist-test-slave-djjw krb5kdc[17524](info): commencing operation
krb5kdc: starting...
W20260928 11:00:53.742156 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:00:53 dist-test-slave-djjw krb5kdc[17524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:00:53.957662 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:35157
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:35157,127.3.252.125:36271,127.3.252.124:45605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:00:54.281497 17537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:54.281818 17537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:54.281966 17537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:54.289296 17537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:00:54.289441 17537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:54.289520 17537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:54.289597 17537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:00:54.289659 17537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:00:54.299832 17537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:35157,127.3.252.125:36271,127.3.252.124:45605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:54.301208 17537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:54.302575 17537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:54.313903 17542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:54.313952 17543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:54.313985 17545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:54.314814 17537 server_base.cc:1034] running on GCE node
I20260928 11:00:54.315619 17537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:54.317050 17537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:54.318260 17537 hybrid_clock.cc:648] HybridClock initialized: now 1790593254318220 us; error 27 us; skew 500 ppm
I20260928 11:00:54.318624 17537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:54 dist-test-slave-djjw krb5kdc[17524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:54.321729 17537 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:00:54.323283 17537 webserver.cc:460] Webserver started at http://127.3.252.126:39263/ using document root <none> and password file <none>
I20260928 11:00:54.323969 17537 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:54.324080 17537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:54.324370 17537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:00:54.327383 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "343da07bb354404dabbe51540183438e"
format_stamp: "Formatted at 2026-09-28 11:00:54 on dist-test-slave-djjw"
server_key: "004be4c775b38558fe4467d45c59ecce"
server_key_iv: "bce8535ddd58a06e0675faedd7ba570e"
server_key_version: "encryptionkey@0"
I20260928 11:00:54.328150 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "343da07bb354404dabbe51540183438e"
format_stamp: "Formatted at 2026-09-28 11:00:54 on dist-test-slave-djjw"
server_key: "004be4c775b38558fe4467d45c59ecce"
server_key_iv: "bce8535ddd58a06e0675faedd7ba570e"
server_key_version: "encryptionkey@0"
I20260928 11:00:54.333227 17537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 11:00:54.336817 17552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:54.338346 17537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:00:54.338557 17537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "343da07bb354404dabbe51540183438e"
format_stamp: "Formatted at 2026-09-28 11:00:54 on dist-test-slave-djjw"
server_key: "004be4c775b38558fe4467d45c59ecce"
server_key_iv: "bce8535ddd58a06e0675faedd7ba570e"
server_key_version: "encryptionkey@0"
I20260928 11:00:54.338753 17537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:54.355315 17537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:54.359308 17537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:54.359577 17537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:54.376767 17537 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:35157
I20260928 11:00:54.376807 17603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:35157 every 8 connection(s)
I20260928 11:00:54.378157 17537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:00:54.381798 17604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:00:54.388582 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17537
I20260928 11:00:54.388782 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:00:54.388556 17604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:00:54.389536 4081 external_mini_cluster.cc:1428] Setting key 2a61ceed5f99af72d46e4dfe7673c6e4
Sep 28 11:00:54 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.125@KRBTEST.COM; defaulting to no policy
I20260928 11:00:54.412333 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.392454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.3.252.126:35157)
0928 11:00:54.396043 (+ 3589us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:54.396049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:54.396077 (+ 28us) server_negotiation.cc:407] Connection header received
0928 11:00:54.396200 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:54.396205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:54.396425 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:54.396595 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:54.397889 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.399069 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.400957 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.401529 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.401669 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:54.405313 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:54.405321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:54.405334 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:54.405364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:54.408108 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:54.408737 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:54.408743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:54.408745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:54.408884 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:54.409238 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:54.409243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:54.409246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:54.409742 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:54.409944 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:54.410474 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 11:00:54.410849 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":3420,"spinlock_wait_cycles":16640,"thread_start_us":3225,"threads_started":1}
I20260928 11:00:54.412302 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.392453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:35157 (local address 127.0.0.1:40852)
0928 11:00:54.393477 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.393524 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:54.395416 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:54.396641 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:54.396648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:54.397334 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:54.397678 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:54.397696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.399217 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.399222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:54.400808 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:54.400822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.401779 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.401786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:54.401981 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:54.402650 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:54.402676 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:54.405081 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:54.408225 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:54.408236 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:54.408248 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:54.408620 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:54.408992 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:54.408997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:54.409001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:54.409143 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:54.409899 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:54.409907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:54.410111 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 11:00:54.410341 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 11:00:54.410980 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":173,"thread_start_us":524,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.125@KRBTEST.COM" created.
I20260928 11:00:54.415635 17604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:00:54.416646 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.416254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47174)
0928 11:00:54.416427 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.416530 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 11:00:54.416920 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
I20260928 11:00:54.418505 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.418158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47186)
0928 11:00:54.418294 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.418366 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 11:00:54.418700 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:36271: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab.
Entry for principal kudu/127.3.252.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab.
I20260928 11:00:54.439227 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:36271
--webserver_interface=127.3.252.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab
--principal=kudu/127.3.252.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:35157,127.3.252.125:36271,127.3.252.124:45605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:00:54.468034 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } attempt: 1
I20260928 11:00:54.469231 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.468770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47198)
0928 11:00:54.468983 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.469094 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 11:00:54.470271 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.469931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47212)
0928 11:00:54.470067 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.470144 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 11:00:54.470448 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:36271: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
I20260928 11:00:54.538720 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } attempt: 2
I20260928 11:00:54.539981 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.539456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47222)
0928 11:00:54.539752 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.539842 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 11:00:54.541038 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.540671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47238)
0928 11:00:54.540820 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.540901 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 11:00:54.541222 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:36271: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
I20260928 11:00:54.632488 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } attempt: 3
I20260928 11:00:54.633666 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.633191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47242)
0928 11:00:54.633402 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.633533 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 11:00:54.634660 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.634313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47250)
0928 11:00:54.634468 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.634534 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 11:00:54.634840 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:36271: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
W20260928 11:00:54.753566 17611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:54.753880 17611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:54.754011 17611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:54.761137 17611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:00:54.761250 17611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:54.761337 17611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:54.761415 17611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:00:54.761475 17611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:00:54.771323 17611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:35157,127.3.252.125:36271,127.3.252.124:45605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab
--principal=kudu/127.3.252.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:36271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.252.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:54.772642 17611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:54.773898 17611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 11:00:54.778092 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } attempt: 4
I20260928 11:00:54.779353 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.778770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47254)
0928 11:00:54.779110 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.779216 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260928 11:00:54.780586 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.780116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47260)
0928 11:00:54.780329 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.780433 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260928 11:00:54.780781 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:36271: Network error: Client connection negotiation failed: client connection to 127.3.252.125:36271: connect: Connection refused (error 111)
W20260928 11:00:54.785578 17616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:54.785547 17619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:54.785650 17617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:54.786300 17611 server_base.cc:1034] running on GCE node
I20260928 11:00:54.787050 17611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:54.788487 17611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:54.789680 17611 hybrid_clock.cc:648] HybridClock initialized: now 1790593254789629 us; error 42 us; skew 500 ppm
I20260928 11:00:54.790030 17611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:54 dist-test-slave-djjw krb5kdc[17524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:54.796931 17611 init.cc:377] Logged in from keytab as kudu/127.3.252.125@KRBTEST.COM (short username kudu)
I20260928 11:00:54.798431 17611 webserver.cc:460] Webserver started at http://127.3.252.125:46845/ using document root <none> and password file <none>
I20260928 11:00:54.799109 17611 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:54.799216 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:54.799501 17611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:00:54.802407 17611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data/instance:
uuid: "e69d7f8e009c4a8f9a84e4becfe71994"
format_stamp: "Formatted at 2026-09-28 11:00:54 on dist-test-slave-djjw"
server_key: "b06ebeb60a4be0da190c63d0becf749d"
server_key_iv: "8746337389a1ce943106c244882f5450"
server_key_version: "encryptionkey@0"
I20260928 11:00:54.803164 17611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/wal/instance:
uuid: "e69d7f8e009c4a8f9a84e4becfe71994"
format_stamp: "Formatted at 2026-09-28 11:00:54 on dist-test-slave-djjw"
server_key: "b06ebeb60a4be0da190c63d0becf749d"
server_key_iv: "8746337389a1ce943106c244882f5450"
server_key_version: "encryptionkey@0"
I20260928 11:00:54.808224 17611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 11:00:54.811450 17626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:54.812702 17611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 11:00:54.812856 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/wal
uuid: "e69d7f8e009c4a8f9a84e4becfe71994"
format_stamp: "Formatted at 2026-09-28 11:00:54 on dist-test-slave-djjw"
server_key: "b06ebeb60a4be0da190c63d0becf749d"
server_key_iv: "8746337389a1ce943106c244882f5450"
server_key_version: "encryptionkey@0"
I20260928 11:00:54.813108 17611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:54.832790 17611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:54.837029 17611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:54.837293 17611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:54.854768 17611 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.125:36271
I20260928 11:00:54.854813 17677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.125:36271 every 8 connection(s)
I20260928 11:00:54.856102 17611 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
I20260928 11:00:54.859156 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17611
I20260928 11:00:54.859354 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-1/wal/instance
I20260928 11:00:54.859900 4081 external_mini_cluster.cc:1428] Setting key 9a44949c2061caf0332649fa94e55eb7
I20260928 11:00:54.860102 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:00:54.868688 17678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.252.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.124@KRBTEST.COM" created.
Sep 28 11:00:54 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:54.889362 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.871332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.3.252.126:35157)
0928 11:00:54.871542 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:54.871548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:54.872689 (+ 1141us) server_negotiation.cc:407] Connection header received
0928 11:00:54.874586 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:54.874592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:54.874671 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:54.874786 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:54.876167 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.877210 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.878761 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.879405 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.879546 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:54.883844 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:54.883850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:54.883853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:54.883889 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:54.886391 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:54.887128 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:54.887133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:54.887135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:54.887251 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:54.887658 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:54.887662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:54.887664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:54.887945 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:54.888082 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:54.888826 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 11:00:54.889055 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 11:00:54.890553 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.871761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:35157 (local address 127.0.0.1:40858)
0928 11:00:54.872516 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.872571 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:54.874331 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:54.874970 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:54.874977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:54.875459 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:54.875995 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:54.876011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.877368 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.877373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:54.878618 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:54.878630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.879637 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.879643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:54.879832 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:54.880526 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:54.880550 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:54.883603 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:54.886539 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:54.886550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:54.886568 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:54.887005 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:54.887356 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:54.887362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:54.887366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:54.887546 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:54.888075 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:54.888084 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:54.888452 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 11:00:54.888687 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 11:00:54.889202 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":318,"threads_started":1}
I20260928 11:00:54.892278 17678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:00:54 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:00:54.907244 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.893044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47262)
0928 11:00:54.893255 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.893274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:54.893383 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:54.894461 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:54.894466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:54.894881 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:54.895059 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:54.895070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.896413 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.896418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:54.897651 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:54.897665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.898590 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.898596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:54.898786 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:54.899364 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:54.899386 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:54.901705 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:54.904765 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:54.904770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:54.904774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:54.905177 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:54.905528 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:54.905533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:54.905537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:54.905678 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:54.906318 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:54.906323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:54.906454 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 11:00:54.906590 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 11:00:54.906866 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 11:00:54.907388 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.893278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.3.252.125:36271)
0928 11:00:54.894047 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:54.894053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:54.894077 (+ 24us) server_negotiation.cc:407] Connection header received
0928 11:00:54.894122 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:54.894128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:54.894302 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:54.894407 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:54.895207 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.896273 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.897781 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:54.898429 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:54.898534 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:54.901919 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:54.901932 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:54.901946 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:54.901981 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:54.904653 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:54.905301 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:54.905308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:54.905310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:54.905420 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:54.905770 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:54.905775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:54.905778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:54.906194 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:54.906376 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:54.906736 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 11:00:54.907027 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":592,"thread_start_us":413,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab.
Entry for principal kudu/127.3.252.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab.
I20260928 11:00:54.909411 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:45605
--webserver_interface=127.3.252.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab
--principal=kudu/127.3.252.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:35157,127.3.252.125:36271,127.3.252.124:45605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:00:54.910105 17678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:00:54.911334 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.910851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38438)
0928 11:00:54.911067 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.911177 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 11:00:54.911681 17627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:54.913197 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.912840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38450)
0928 11:00:54.912978 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.913056 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 11:00:54.913419 17678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:54.962745 17678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 1
I20260928 11:00:54.963932 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.963495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38460)
0928 11:00:54.963687 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.963787 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 11:00:54.964918 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:54.964570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38476)
0928 11:00:54.964745 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:54.964813 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 11:00:54.965116 17678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:55.033329 17678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 2
I20260928 11:00:55.034545 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.034079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38484)
0928 11:00:55.034298 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.034389 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 11:00:55.035569 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.035226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38488)
0928 11:00:55.035376 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.035453 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 11:00:55.035777 17678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:55.080054 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } attempt: 5
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:00:55.094498 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.080811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47274)
0928 11:00:55.081030 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.081051 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:55.081159 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:55.081493 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:55.081497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:55.081905 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:55.082092 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.082101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.083523 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.083528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.084381 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.084393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.085316 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.085321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.085518 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.086190 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:55.086209 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:55.088174 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:55.092056 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.092061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.092063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.092457 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.092867 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.092871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.092873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.093016 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.093647 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:55.093651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:55.093793 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 11:00:55.093948 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 11:00:55.094158 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 11:00:55.094689 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.080930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.3.252.125:36271)
0928 11:00:55.081191 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.081197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.081219 (+ 22us) server_negotiation.cc:407] Connection header received
0928 11:00:55.081281 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.081286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.081362 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.081466 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.082253 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.083343 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.084526 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.085134 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.085262 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.088369 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.088376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.088381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.088421 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.091872 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.092580 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.092586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.092590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.092743 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.093121 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.093126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.093129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.093522 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.093679 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.094100 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.094369 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 11:00:55.095806 17604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:00:55.096985 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.096581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38502)
0928 11:00:55.096767 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.096844 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 11:00:55.097949 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.097582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38516)
0928 11:00:55.097722 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.097801 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 11:00:55.098126 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:55.126974 17678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 3
I20260928 11:00:55.128014 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.127565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38518)
0928 11:00:55.127775 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.127867 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 11:00:55.129024 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.128645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38526)
0928 11:00:55.128843 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.128912 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260928 11:00:55.129225 17678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:55.149341 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 1
I20260928 11:00:55.150482 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.150040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38540)
0928 11:00:55.150219 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.150322 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 11:00:55.151355 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.151056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38554)
0928 11:00:55.151191 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.151257 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 11:00:55.151542 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:55.219733 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 2
I20260928 11:00:55.220943 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.220332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38566)
0928 11:00:55.220600 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.220715 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 11:00:55.221958 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.221539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38576)
0928 11:00:55.221689 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.221772 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 11:00:55.222211 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
W20260928 11:00:55.228062 17685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:55.228397 17685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:55.228511 17685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:55.235813 17685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:00:55.235916 17685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:55.235988 17685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:55.236057 17685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:00:55.236114 17685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:00:55.246017 17685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:35157,127.3.252.125:36271,127.3.252.124:45605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab
--principal=kudu/127.3.252.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:45605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.252.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:55.247303 17685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:55.248636 17685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:55.260142 17690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:55.260114 17693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:55.260085 17691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:55.263288 17685 server_base.cc:1034] running on GCE node
I20260928 11:00:55.264153 17685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:55.265602 17685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:55.266786 17685 hybrid_clock.cc:648] HybridClock initialized: now 1790593255266743 us; error 39 us; skew 500 ppm
I20260928 11:00:55.267130 17685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:55.270061 17685 init.cc:377] Logged in from keytab as kudu/127.3.252.124@KRBTEST.COM (short username kudu)
I20260928 11:00:55.271580 17685 webserver.cc:460] Webserver started at http://127.3.252.124:44629/ using document root <none> and password file <none>
I20260928 11:00:55.272281 17685 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:55.272382 17685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:55.272411 17678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 4
I20260928 11:00:55.272707 17685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:00:55.273427 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.272985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38592)
0928 11:00:55.273195 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.273275 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 11:00:55.274415 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.274090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38594)
0928 11:00:55.274227 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.274298 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 11:00:55.274613 17678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:55.275844 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data/instance:
uuid: "02667a806ee647158398220ea79c84e5"
format_stamp: "Formatted at 2026-09-28 11:00:55 on dist-test-slave-djjw"
server_key: "be8b56c78c42e089b0cbbf0d910bb31e"
server_key_iv: "03e302864c9a5ab1033778f9e00782b6"
server_key_version: "encryptionkey@0"
I20260928 11:00:55.276631 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/wal/instance:
uuid: "02667a806ee647158398220ea79c84e5"
format_stamp: "Formatted at 2026-09-28 11:00:55 on dist-test-slave-djjw"
server_key: "be8b56c78c42e089b0cbbf0d910bb31e"
server_key_iv: "03e302864c9a5ab1033778f9e00782b6"
server_key_version: "encryptionkey@0"
I20260928 11:00:55.281724 17685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 11:00:55.285032 17700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:55.286334 17685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:00:55.286489 17685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/wal
uuid: "02667a806ee647158398220ea79c84e5"
format_stamp: "Formatted at 2026-09-28 11:00:55 on dist-test-slave-djjw"
server_key: "be8b56c78c42e089b0cbbf0d910bb31e"
server_key_iv: "03e302864c9a5ab1033778f9e00782b6"
server_key_version: "encryptionkey@0"
I20260928 11:00:55.286662 17685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:55.309686 17685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:55.313652 17685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:55.313946 17685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:55.328428 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 3
I20260928 11:00:55.329559 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.329069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38606)
0928 11:00:55.329281 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.329407 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 11:00:55.330704 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.330233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38620)
0928 11:00:55.330445 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.330563 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 11:00:55.330982 17604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:45605: Network error: Client connection negotiation failed: client connection to 127.3.252.124:45605: connect: Connection refused (error 111)
I20260928 11:00:55.331071 17685 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.124:45605
I20260928 11:00:55.331095 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.124:45605 every 8 connection(s)
I20260928 11:00:55.332332 17685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
I20260928 11:00:55.335935 17752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:00:55.340186 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17685
I20260928 11:00:55.340344 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790593056029081-4081-0/minicluster-data/master-2/wal/instance
I20260928 11:00:55.340759 4081 external_mini_cluster.cc:1428] Setting key 94a17ceda668caa39ae19527bb219934
I20260928 11:00:55.342671 17752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
I20260928 11:00:55.359687 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.345664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.3.252.126:35157)
0928 11:00:55.345888 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.345894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.346584 (+ 690us) server_negotiation.cc:407] Connection header received
0928 11:00:55.346832 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.346836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.346895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.346974 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.348182 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.348842 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.350286 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.350700 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.350808 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.354050 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.354057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.354062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.354096 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.356707 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.357496 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.357500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.357502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.357623 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.358073 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.358077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.358079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.358315 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.358450 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.359115 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.359329 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 11:00:55.360695 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.345671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:35157 (local address 127.0.0.1:40860)
0928 11:00:55.346440 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.346496 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:55.346703 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:55.347054 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:55.347060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:55.347617 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:55.348017 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.348034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.348987 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.348992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.350154 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.350167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.350918 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.350925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.351058 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.351680 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:55.351705 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:55.353862 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:55.356832 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.356844 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.356859 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.357373 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.357723 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.357728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.357732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.357959 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.358435 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:55.358443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:55.358760 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 11:00:55.359005 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 11:00:55.359458 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":395,"threads_started":1}
I20260928 11:00:55.360716 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.342537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.3.252.124:45605)
0928 11:00:55.343345 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.343359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.343432 (+ 73us) server_negotiation.cc:407] Connection header received
0928 11:00:55.345604 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.345625 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.346102 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.346443 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.347892 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.349051 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.350388 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.350890 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.351052 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.354909 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.354917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.354928 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.354959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.357470 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.358131 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.358137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.358139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.358253 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.358613 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.358616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.358618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.358980 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.359231 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.359573 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.359807 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":565,"spinlock_wait_cycles":15232,"thread_start_us":326,"threads_started":1}
I20260928 11:00:55.362381 17752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:55.376550 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.363229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:36271 (local address 127.0.0.1:47286)
0928 11:00:55.363382 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.363400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:55.363479 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:55.363962 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:55.363965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:55.364275 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:55.364426 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.364434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.365782 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.365788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.367156 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.367168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.368128 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.368133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.368295 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.368840 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:55.368857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:55.370850 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:55.374036 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.374041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.374045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.374499 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.374880 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.374883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.374886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.375060 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.375563 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:55.375568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:55.375732 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 11:00:55.375891 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 11:00:55.376179 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 11:00:55.376556 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.363236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.3.252.125:36271)
0928 11:00:55.363421 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.363426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.363446 (+ 20us) server_negotiation.cc:407] Connection header received
0928 11:00:55.363607 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.363614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.363694 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.363787 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.364580 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.365637 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.367294 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.367967 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.368098 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.371060 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.371069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.371074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.371118 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.373884 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.374616 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.374622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.374624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.374761 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.375168 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.375171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.375173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.375449 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.375565 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.375975 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.376215 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 11:00:55.377790 17752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:00:55.379731 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.363768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.3.252.126:35157)
0928 11:00:55.363917 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.363922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.363968 (+ 46us) server_negotiation.cc:407] Connection header received
0928 11:00:55.364135 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.364140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.364201 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.364274 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.365782 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.366943 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.368562 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.368993 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.369098 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.373532 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.373537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.373540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.373575 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.376513 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.377375 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.377381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.377384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.377567 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.378041 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.378046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.378050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.378392 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.378665 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.379159 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.379381 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:00:55.390558 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.378516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38636)
0928 11:00:55.378766 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.378784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:55.378883 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:55.379281 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:55.379286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:55.379717 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:55.379882 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.379894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.381018 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.381024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.382222 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.382233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.382859 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.382864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.383055 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.383552 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:55.383568 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:55.385375 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:55.388396 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.388401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.388405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.388724 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.389028 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.389033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.389036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.389153 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.389615 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:55.389619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:55.389785 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 11:00:55.389962 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 11:00:55.390285 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 11:00:55.390823 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.378616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.3.252.124:45605)
0928 11:00:55.378768 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.378773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.378829 (+ 56us) server_negotiation.cc:407] Connection header received
0928 11:00:55.378975 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.378980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.379051 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.379131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.380036 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.380912 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.382337 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.382736 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.382838 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.385543 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.385547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.385551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.385578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.388286 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.388822 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.388827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.388830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.388931 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.389244 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.389249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.389252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.389510 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.389633 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.390060 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.390462 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":49}
I20260928 11:00:55.395450 17752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: Bootstrap starting.
I20260928 11:00:55.396221 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.381339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.3.252.125:36271)
0928 11:00:55.381490 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.381496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.381549 (+ 53us) server_negotiation.cc:407] Connection header received
0928 11:00:55.381707 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.381712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.381781 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.381912 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.383446 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.384501 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.386318 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.386922 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.387285 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.391309 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.391316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.391321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.391358 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.393731 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.394359 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.394364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.394366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.394499 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.394898 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.394903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.394907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.395173 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.395429 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.395705 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.395925 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 11:00:55.399067 17752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:55.400183 17752 log.cc:824] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:55.400981 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:00:55.403034 17752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: No bootstrap required, opened a new log
I20260928 11:00:55.413336 17752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.414001 17752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:55.414188 17752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02667a806ee647158398220ea79c84e5, State: Initialized, Role: FOLLOWER
I20260928 11:00:55.415066 17752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.422377 17765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.424981 17765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:00:55.424853 17752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:00:55.428228 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.406380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.3.252.124:45605)
0928 11:00:55.409376 (+ 2996us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.409381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.409894 (+ 513us) server_negotiation.cc:407] Connection header received
0928 11:00:55.410132 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.410137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.410207 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.410265 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.413096 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.414205 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.416366 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.417119 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.417204 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.421132 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.421139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.421144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.421186 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.424680 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.425721 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.425726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.425730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.425953 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.426440 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.426444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.426447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.426843 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.427055 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.427658 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.427865 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2872}
I20260928 11:00:55.428249 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.405628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.3.252.125:36271)
0928 11:00:55.405798 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.405804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.409384 (+ 3580us) server_negotiation.cc:407] Connection header received
0928 11:00:55.410037 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.410043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.410123 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.410188 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.412762 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.413797 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.416051 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.416736 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.416835 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.421257 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.421266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.421271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.421320 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.424510 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.425603 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.425608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.425611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.425773 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.426277 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.426281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.426284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.426675 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.426913 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.427580 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.427851 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 11:00:55.430619 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.406676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.3.252.126:35157)
0928 11:00:55.406851 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.406857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.412721 (+ 5864us) server_negotiation.cc:407] Connection header received
0928 11:00:55.413273 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.413279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.413360 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.413429 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.415906 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.416940 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.419503 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.420163 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.420267 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.424129 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.424136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.424140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.424196 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.427493 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.428295 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.428301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.428304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.428470 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.428933 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.428938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.428941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.429293 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.429511 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.430067 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.430295 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 11:00:55.441630 17776 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 11:00:55.441807 17776 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 11:00:55.439258 17685 master_runner.cc:186] Error getting master registration for 127.3.252.126:35157: OK, instance_id { permanent_uuid: "343da07bb354404dabbe51540183438e" instance_seqno: 1790593254371867 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 11:00:55.450057 17685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 11:00:55.508206 17604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 4
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
I20260928 11:00:55.523617 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.508867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.3.252.124:45605)
0928 11:00:55.509020 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.509024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.509042 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:00:55.509210 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.509213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.509291 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.509386 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.510367 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.511224 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.513003 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.513498 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.513599 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.517242 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.517248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.517251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.517284 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.521067 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.521805 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.521809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.521812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.522001 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.522369 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.522373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.522375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.522660 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.522763 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.523202 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.523367 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 11:00:55.523772 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.508744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38652)
0928 11:00:55.508956 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.508978 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:55.509103 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:55.509427 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:55.509432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:55.509982 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:55.510187 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.510200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.511358 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.511364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.512835 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.512852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.513647 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.513654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.513883 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.514479 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:55.514503 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:55.517071 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:55.521190 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.521196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.521200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.521698 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.522102 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.522107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.522111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.522273 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.522774 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:55.522780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:55.522955 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 11:00:55.523111 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 11:00:55.523400 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 11:00:55.529629 17604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: Bootstrap starting.
I20260928 11:00:55.533558 17604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:55.534837 17604 log.cc:824] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:55.538820 17604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: No bootstrap required, opened a new log
I20260928 11:00:55.549484 17604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.549911 17604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:55.550041 17604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343da07bb354404dabbe51540183438e, State: Initialized, Role: FOLLOWER
I20260928 11:00:55.550707 17604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.552170 17779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.552546 17779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:00:55.553596 17604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [sys.catalog]: configured and running, proceeding with master startup.
W20260928 11:00:55.562573 17790 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 11:00:55.562780 17790 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 11:00:55.566035 17706 tablet.cc:2369] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:00:55.573861 17678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } attempt: 5
I20260928 11:00:55.562176 17537 master_runner.cc:186] Error getting master registration for 127.3.252.125:36271: OK, instance_id { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" instance_seqno: 1790593254849869 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 11:00:55.574586 17537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 11:00:55 dist-test-slave-djjw krb5kdc[17524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
I20260928 11:00:55.590786 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.574499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:45605 (local address 127.0.0.1:38662)
0928 11:00:55.574847 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 11:00:55.574869 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 11:00:55.574983 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:00:55.575425 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:00:55.575430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:00:55.575961 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:00:55.576171 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.576184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.577617 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.577623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.579081 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:00:55.579097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.580106 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.580112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:00:55.580337 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.581019 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:00:55.581043 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:00:55.583779 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:00:55.587819 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.587825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.587829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.588341 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.588820 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:00:55.588826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:00:55.588830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:00:55.588987 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:00:55.589733 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:00:55.589737 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:00:55.589932 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 11:00:55.590087 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 11:00:55.590382 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260928 11:00:55.591859 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:55.574923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.3.252.124:45605)
0928 11:00:55.575104 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:55.575110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:55.575132 (+ 22us) server_negotiation.cc:407] Connection header received
0928 11:00:55.575189 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:55.575194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:55.575267 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:55.575368 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:55.576327 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.577455 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.579232 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:55.579925 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:55.580058 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:55.583995 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:55.584002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:55.584007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:55.584060 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:55.587658 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.588531 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.588537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.588541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.588678 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:55.589118 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:55.589122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:55.589125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:55.589525 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:55.589672 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:55.591233 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0928 11:00:55.591491 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:00:55.598026 17678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994: Bootstrap starting.
I20260928 11:00:55.602200 17678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:55.603561 17678 log.cc:824] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:55.608786 17678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994: No bootstrap required, opened a new log
W20260928 11:00:55.612406 17558 tablet.cc:2369] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:00:55.621201 17678 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.621631 17678 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:55.621788 17678 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69d7f8e009c4a8f9a84e4becfe71994, State: Initialized, Role: FOLLOWER
I20260928 11:00:55.622503 17678 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.624852 17678 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:00:55.629647 17793 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.630896 17793 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:00:55.631270 17793 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:00:55.636263 17793 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.640093 17579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343da07bb354404dabbe51540183438e" is_pre_election: true
I20260928 11:00:55.640693 17579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69d7f8e009c4a8f9a84e4becfe71994 in term 0.
I20260928 11:00:55.641494 17629 leader_election.cc:302] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343da07bb354404dabbe51540183438e, e69d7f8e009c4a8f9a84e4becfe71994; no voters:
I20260928 11:00:55.643389 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02667a806ee647158398220ea79c84e5" is_pre_election: true
I20260928 11:00:55.643594 17765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:00:55.643903 17765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.644397 17793 leader_election.cc:288] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343da07bb354404dabbe51540183438e (127.3.252.126:35157), 02667a806ee647158398220ea79c84e5 (127.3.252.124:45605)
I20260928 11:00:55.644589 17793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:00:55.644707 17793 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:00:55.644796 17793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:55.645344 17765 leader_election.cc:288] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343da07bb354404dabbe51540183438e (127.3.252.126:35157), e69d7f8e009c4a8f9a84e4becfe71994 (127.3.252.125:36271)
I20260928 11:00:55.646763 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02667a806ee647158398220ea79c84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" is_pre_election: true
I20260928 11:00:55.647579 17727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69d7f8e009c4a8f9a84e4becfe71994 in term 0.
I20260928 11:00:55.649618 17579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02667a806ee647158398220ea79c84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343da07bb354404dabbe51540183438e" is_pre_election: true
I20260928 11:00:55.649909 17579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02667a806ee647158398220ea79c84e5 in term 0.
I20260928 11:00:55.649757 17793 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.650252 17653 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02667a806ee647158398220ea79c84e5 in current term 1: Already voted for candidate e69d7f8e009c4a8f9a84e4becfe71994 in this term.
I20260928 11:00:55.650437 17703 leader_election.cc:302] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02667a806ee647158398220ea79c84e5, 343da07bb354404dabbe51540183438e; no voters:
I20260928 11:00:55.650997 17765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:00:55.651131 17765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:00:55.651214 17765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:55.654398 17765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.654719 17579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343da07bb354404dabbe51540183438e"
I20260928 11:00:55.654980 17579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:55.655551 17578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02667a806ee647158398220ea79c84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343da07bb354404dabbe51540183438e"
I20260928 11:00:55.655949 17765 leader_election.cc:288] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [CANDIDATE]: Term 1 election: Requested vote from peers 343da07bb354404dabbe51540183438e (127.3.252.126:35157), e69d7f8e009c4a8f9a84e4becfe71994 (127.3.252.125:36271)
I20260928 11:00:55.656284 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02667a806ee647158398220ea79c84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69d7f8e009c4a8f9a84e4becfe71994"
I20260928 11:00:55.656512 17653 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 02667a806ee647158398220ea79c84e5 in current term 1: Already voted for candidate e69d7f8e009c4a8f9a84e4becfe71994 in this term.
I20260928 11:00:55.659068 17702 leader_election.cc:302] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02667a806ee647158398220ea79c84e5; no voters: 343da07bb354404dabbe51540183438e, e69d7f8e009c4a8f9a84e4becfe71994
I20260928 11:00:55.663214 17579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69d7f8e009c4a8f9a84e4becfe71994 in term 1.
I20260928 11:00:55.663424 17765 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 11:00:55.665592 17629 leader_election.cc:302] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343da07bb354404dabbe51540183438e, e69d7f8e009c4a8f9a84e4becfe71994; no voters:
I20260928 11:00:55.665990 17794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:00:55.666571 17793 leader_election.cc:288] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [CANDIDATE]: Term 1 election: Requested vote from peers 343da07bb354404dabbe51540183438e (127.3.252.126:35157), 02667a806ee647158398220ea79c84e5 (127.3.252.124:45605)
I20260928 11:00:55.667048 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02667a806ee647158398220ea79c84e5"
I20260928 11:00:55.667342 17727 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e69d7f8e009c4a8f9a84e4becfe71994 in current term 1: Already voted for candidate 02667a806ee647158398220ea79c84e5 in this term.
I20260928 11:00:55.678115 17794 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [term 1 LEADER]: Becoming Leader. State: Replica: e69d7f8e009c4a8f9a84e4becfe71994, State: Running, Role: LEADER
I20260928 11:00:55.678874 17794 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } }
I20260928 11:00:55.691776 17793 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69d7f8e009c4a8f9a84e4becfe71994. Latest consensus state: current_term: 1 leader_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.692126 17793 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:55.702507 17808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:00:55.715314 17808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:00:55.735459 17579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [term 1 FOLLOWER]: Refusing update from remote peer e69d7f8e009c4a8f9a84e4becfe71994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:00:55.736150 17727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [term 1 FOLLOWER]: Refusing update from remote peer e69d7f8e009c4a8f9a84e4becfe71994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:00:55.736541 17793 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:00:55.737811 17793 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:00:55.774475 17779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69d7f8e009c4a8f9a84e4becfe71994. Latest consensus state: current_term: 1 leader_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.774776 17779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:00:55.778666 17810 mvcc.cc:205] Tried to move back new op lower bound from 7334269975477321728 to 7334269975266930688. Current Snapshot: MvccSnapshot[applied={T|T < 7334269975477321728}]
I20260928 11:00:55.780702 17812 mvcc.cc:205] Tried to move back new op lower bound from 7334269975477321728 to 7334269975266930688. Current Snapshot: MvccSnapshot[applied={T|T < 7334269975477321728}]
I20260928 11:00:55.782027 17765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69d7f8e009c4a8f9a84e4becfe71994. Latest consensus state: current_term: 1 leader_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.782358 17765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:00:55.784938 17779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.785181 17779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:00:55.787075 17793 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.787336 17793 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:55.787683 17802 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.788084 17802 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994 [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:55.788563 17811 mvcc.cc:205] Tried to move back new op lower bound from 7334269975477321728 to 7334269975266930688. Current Snapshot: MvccSnapshot[applied={T|T < 7334269975477321728}]
I20260928 11:00:55.788785 17808 catalog_manager.cc:1347] Generated new cluster ID: 68031130fd6145d0b3d4c304eac07772
I20260928 11:00:55.788887 17808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:00:55.813634 17765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343da07bb354404dabbe51540183438e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 35157 } } peers { permanent_uuid: "e69d7f8e009c4a8f9a84e4becfe71994" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 36271 } } peers { permanent_uuid: "02667a806ee647158398220ea79c84e5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 45605 } } }
I20260928 11:00:55.814975 17765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:00:55.824306 17808 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:00:55.826472 17808 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:00:55.852278 17808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69d7f8e009c4a8f9a84e4becfe71994: Generated new TSK 0
I20260928 11:00:55.853587 17808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:00:56.444231 17776 catalog_manager.cc:1259] Loaded cluster ID: 68031130fd6145d0b3d4c304eac07772
I20260928 11:00:56.444344 17776 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: loading cluster ID for follower catalog manager: success
I20260928 11:00:56.447013 17776 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: acquiring CA information for follower catalog manager: success
I20260928 11:00:56.449014 17776 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02667a806ee647158398220ea79c84e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 11:00:56.564793 17790 catalog_manager.cc:1259] Loaded cluster ID: 68031130fd6145d0b3d4c304eac07772
I20260928 11:00:56.564898 17790 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: loading cluster ID for follower catalog manager: success
I20260928 11:00:56.567351 17790 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: acquiring CA information for follower catalog manager: success
I20260928 11:00:56.569124 17790 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 343da07bb354404dabbe51540183438e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 11:00:57.029624 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17537
I20260928 11:00:57.030829 17599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:57.139891 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17537
I20260928 11:00:57.157385 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17611
I20260928 11:00:57.158463 17673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:57.267721 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17611
I20260928 11:00:57.284649 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17685
I20260928 11:00:57.285744 17747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:57.397984 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17685
2026-09-28T11:00:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4662 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:00:57 dist-test-slave-djjw krb5kdc[17828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:00:57 dist-test-slave-djjw krb5kdc[17828](info): set up 2 sockets
Sep 28 11:00:57 dist-test-slave-djjw krb5kdc[17828](info): commencing operation
krb5kdc: starting...
W20260928 11:00:58.410296 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:00:58 dist-test-slave-djjw krb5kdc[17828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:00:58.629444 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41331
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:00:58.942852 17841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:00:58.943158 17841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:00:58.943288 17841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:00:58.950505 17841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:00:58.950624 17841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:00:58.950691 17841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:00:58.950764 17841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:00:58.950825 17841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:00:58.960506 17841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:41331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:00:58.961771 17841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:00:58.963037 17841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:00:58.974329 17846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:58.974289 17847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:00:58.974375 17849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:00:58.979414 17841 server_base.cc:1034] running on GCE node
I20260928 11:00:58.980201 17841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:00:58.981585 17841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:00:58.982796 17841 hybrid_clock.cc:648] HybridClock initialized: now 1790593258982738 us; error 45 us; skew 500 ppm
I20260928 11:00:58.983143 17841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:00:58 dist-test-slave-djjw krb5kdc[17828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593258, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:00:58.986198 17841 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:00:58.987687 17841 webserver.cc:460] Webserver started at http://127.3.252.126:34591/ using document root <none> and password file <none>
I20260928 11:00:58.988390 17841 fs_manager.cc:359] Metadata directory not provided
I20260928 11:00:58.988507 17841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:00:58.988799 17841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:00:58.991719 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "3cc448c962ea4d21846d033439515707"
format_stamp: "Formatted at 2026-09-28 11:00:58 on dist-test-slave-djjw"
server_key: "13a3f79171bbfba1102e4f185f409923"
server_key_iv: "c5e43dc2d70270ec149d555cb7f4fddd"
server_key_version: "encryptionkey@0"
I20260928 11:00:58.992424 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "3cc448c962ea4d21846d033439515707"
format_stamp: "Formatted at 2026-09-28 11:00:58 on dist-test-slave-djjw"
server_key: "13a3f79171bbfba1102e4f185f409923"
server_key_iv: "c5e43dc2d70270ec149d555cb7f4fddd"
server_key_version: "encryptionkey@0"
I20260928 11:00:58.997449 17841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 11:00:59.000902 17856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:00:59.002583 17841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 11:00:59.002758 17841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "3cc448c962ea4d21846d033439515707"
format_stamp: "Formatted at 2026-09-28 11:00:58 on dist-test-slave-djjw"
server_key: "13a3f79171bbfba1102e4f185f409923"
server_key_iv: "c5e43dc2d70270ec149d555cb7f4fddd"
server_key_version: "encryptionkey@0"
I20260928 11:00:59.002946 17841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:00:59.015357 17841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:00:59.019711 17841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:00:59.019971 17841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:00:59.037258 17841 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:41331
I20260928 11:00:59.037292 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:41331 every 8 connection(s)
I20260928 11:00:59.038671 17841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:00:59.039805 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17841
I20260928 11:00:59.040028 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:00:59.040827 4081 external_mini_cluster.cc:1428] Setting key 3989ddbb5b91d18b3a046532756ab309
I20260928 11:00:59.043311 17908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 11:00:59 dist-test-slave-djjw krb5kdc[17828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:00:59.056227 17908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707: Bootstrap starting.
I20260928 11:00:59.059379 17908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707: Neither blocks nor log segments found. Creating new log.
I20260928 11:00:59.060608 17908 log.cc:824] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707: Log is configured to *not* fsync() on all Append() calls
I20260928 11:00:59.063364 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:00:59.042680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.3.252.126:41331)
0928 11:00:59.043458 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0928 11:00:59.043471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 11:00:59.043512 (+ 41us) server_negotiation.cc:407] Connection header received
0928 11:00:59.045117 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:00:59.045138 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:00:59.045518 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:00:59.045882 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:00:59.047617 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:59.048948 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:59.050789 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:00:59.051437 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:00:59.051566 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:00:59.056354 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:00:59.056365 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:00:59.056383 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:00:59.056420 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:00:59.059280 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:59.060163 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:59.060169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:59.060175 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:59.060341 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:00:59.060739 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:00:59.060744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:00:59.060747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:00:59.061371 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:00:59.061666 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 11:00:59.062045 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 11:00:59.062424 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":347,"threads_started":1}
I20260928 11:00:59.063644 17908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707: No bootstrap required, opened a new log
I20260928 11:00:59.066915 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:00:59.072254 17908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc448c962ea4d21846d033439515707" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41331 } }
I20260928 11:00:59.072592 17908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:00:59.072659 17908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc448c962ea4d21846d033439515707, State: Initialized, Role: FOLLOWER
I20260928 11:00:59.073130 17908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc448c962ea4d21846d033439515707" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41331 } }
I20260928 11:00:59.073315 17908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:00:59.073406 17908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:00:59.073525 17908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:00:59.075618 17908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc448c962ea4d21846d033439515707" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41331 } }
I20260928 11:00:59.076052 17908 leader_election.cc:302] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cc448c962ea4d21846d033439515707; no voters:
I20260928 11:00:59.076556 17908 leader_election.cc:288] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:00:59.076680 17913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:00:59.077262 17913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc448c962ea4d21846d033439515707, State: Running, Role: LEADER
I20260928 11:00:59.077737 17913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc448c962ea4d21846d033439515707" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41331 } }
I20260928 11:00:59.078449 17908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:00:59.080534 17915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cc448c962ea4d21846d033439515707. Latest consensus state: current_term: 1 leader_uuid: "3cc448c962ea4d21846d033439515707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc448c962ea4d21846d033439515707" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41331 } } }
I20260928 11:00:59.080565 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cc448c962ea4d21846d033439515707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc448c962ea4d21846d033439515707" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41331 } } }
I20260928 11:00:59.080904 17915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:59.080905 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707 [sys.catalog]: This master's current role is: LEADER
I20260928 11:00:59.083026 17921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:00:59.088258 17921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:00:59.096812 17921 catalog_manager.cc:1347] Generated new cluster ID: 10a07386d6ed4b16a2e3c3ccd1d831e6
I20260928 11:00:59.096937 17921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:00:59.132874 17921 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:00:59.134326 17921 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:00:59.148336 17921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cc448c962ea4d21846d033439515707: Generated new TSK 0
I20260928 11:00:59.149245 17921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:00:59.179451 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17841
I20260928 11:00:59.180648 17903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:00:59.276510 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17841
2026-09-28T11:00:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1873 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:00:59 dist-test-slave-djjw krb5kdc[17936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:00:59 dist-test-slave-djjw krb5kdc[17936](info): set up 2 sockets
Sep 28 11:00:59 dist-test-slave-djjw krb5kdc[17936](info): commencing operation
krb5kdc: starting...
W20260928 11:01:00.290266 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:01:00 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:01:00.513196 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:44131
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:44131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:00.832911 17949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:00.833210 17949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:00.833343 17949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:00.840574 17949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:01:00.840698 17949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:00.840785 17949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:00.840859 17949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:01:00.840920 17949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:01:00.850903 17949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:44131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:00.852254 17949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:00.853531 17949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:00.864886 17957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:00.864938 17955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:00.864986 17954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:00.865289 17949 server_base.cc:1034] running on GCE node
I20260928 11:01:00.866274 17949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:00.867599 17949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:00.868794 17949 hybrid_clock.cc:648] HybridClock initialized: now 1790593260868743 us; error 38 us; skew 500 ppm
I20260928 11:01:00.869118 17949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:00 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:00.876252 17949 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:01:00.877800 17949 webserver.cc:460] Webserver started at http://127.3.252.126:34627/ using document root <none> and password file <none>
I20260928 11:01:00.878549 17949 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:00.878664 17949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:00.878948 17949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:00.881946 17949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "7109a2c4793a4783bc3baafa4cfbc54e"
format_stamp: "Formatted at 2026-09-28 11:01:00 on dist-test-slave-djjw"
server_key: "cdf6f46a18553ed31cfbf92523f59f6e"
server_key_iv: "4280da3b5e2ae4396274dc5b13e8e8aa"
server_key_version: "encryptionkey@0"
I20260928 11:01:00.882681 17949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "7109a2c4793a4783bc3baafa4cfbc54e"
format_stamp: "Formatted at 2026-09-28 11:01:00 on dist-test-slave-djjw"
server_key: "cdf6f46a18553ed31cfbf92523f59f6e"
server_key_iv: "4280da3b5e2ae4396274dc5b13e8e8aa"
server_key_version: "encryptionkey@0"
I20260928 11:01:00.887706 17949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 11:01:00.891006 17964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:00.892254 17949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:01:00.892409 17949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "7109a2c4793a4783bc3baafa4cfbc54e"
format_stamp: "Formatted at 2026-09-28 11:01:00 on dist-test-slave-djjw"
server_key: "cdf6f46a18553ed31cfbf92523f59f6e"
server_key_iv: "4280da3b5e2ae4396274dc5b13e8e8aa"
server_key_version: "encryptionkey@0"
I20260928 11:01:00.892588 17949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:00.907408 17949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:00.911469 17949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:00.911731 17949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:00.928285 17949 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:44131
I20260928 11:01:00.928308 18015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:44131 every 8 connection(s)
I20260928 11:01:00.929579 17949 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:01:00.933063 18016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:00.933929 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 17949
I20260928 11:01:00.934145 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:01:00.934939 4081 external_mini_cluster.cc:1428] Setting key e7dcde40327f14f936d1d30f09dfb544
I20260928 11:01:00.945330 18016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Bootstrap starting.
I20260928 11:01:00.949093 18016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Neither blocks nor log segments found. Creating new log.
Sep 28 11:01:00 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:00.950320 18016 log.cc:824] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:00.953574 18016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: No bootstrap required, opened a new log
I20260928 11:01:00.958680 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:00.937962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.3.252.126:44131)
0928 11:01:00.938718 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:00.938732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:00.938770 (+ 38us) server_negotiation.cc:407] Connection header received
0928 11:01:00.940377 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:00.940399 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:00.940774 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:00.941074 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:00.942802 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:00.944105 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:00.945957 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:00.946616 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:00.946742 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:00.951416 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:00.951426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:00.951445 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:00.951495 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:00.954680 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:00.955463 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:00.955468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:00.955474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:00.955651 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:00.956023 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:00.956026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:00.956028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:00.956573 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:00.956810 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:00.957220 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 11:01:00.957602 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":341,"threads_started":1}
I20260928 11:01:00.961910 18016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:00.962277 18016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:00.962365 18016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7109a2c4793a4783bc3baafa4cfbc54e, State: Initialized, Role: FOLLOWER
I20260928 11:01:00.963050 18016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:00.963286 18016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:01:00.963413 18016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:01:00.963588 18016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:00.965814 18016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:00.966351 18016 leader_election.cc:302] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7109a2c4793a4783bc3baafa4cfbc54e; no voters:
I20260928 11:01:00.966940 18016 leader_election.cc:288] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:01:00.967119 18021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:01:00.968130 18021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 LEADER]: Becoming Leader. State: Replica: 7109a2c4793a4783bc3baafa4cfbc54e, State: Running, Role: LEADER
I20260928 11:01:00.968633 18021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:00.969007 18016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:01:00.971041 18023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7109a2c4793a4783bc3baafa4cfbc54e. Latest consensus state: current_term: 1 leader_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } } }
I20260928 11:01:00.971453 18023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:00.972038 18022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } } }
I20260928 11:01:00.972333 18022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:00.972371 18030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:01:00.981431 18030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:01:00.990530 18030 catalog_manager.cc:1347] Generated new cluster ID: 03cfc9dd1c0545cb9535c0ea82c4b12c
I20260928 11:01:00.990656 18030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:01:00.999960 18030 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:01:01.001256 18030 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:01:01.013676 18030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Generated new TSK 0
I20260928 11:01:01.014613 18030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 11:01:01.067071 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:44131
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:01.385790 18043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:01.386200 18043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:01.386301 18043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:01.393522 18043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:01.393637 18043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:01.393748 18043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:01:01.403529 18043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:01.404755 18043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:01.406064 18043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:01.418977 18048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:01.418902 18049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:01.418924 18051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:01.419759 18043 server_base.cc:1034] running on GCE node
I20260928 11:01:01.420331 18043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:01.421262 18043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:01.422482 18043 hybrid_clock.cc:648] HybridClock initialized: now 1790593261422402 us; error 52 us; skew 500 ppm
I20260928 11:01:01.422783 18043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:01 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:01.425730 18043 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:01:01.427212 18043 webserver.cc:460] Webserver started at http://127.3.252.65:36285/ using document root <none> and password file <none>
I20260928 11:01:01.427901 18043 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:01.428035 18043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:01.428347 18043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:01.431185 18043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "40eba88f0d4f4604b64f9c9689c1e1fd"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "6a36b06abf6d77db9322fe3f607dd477"
server_key_iv: "e529ce8fed32c790888b4bac94525506"
server_key_version: "encryptionkey@0"
I20260928 11:01:01.431849 18043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "40eba88f0d4f4604b64f9c9689c1e1fd"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "6a36b06abf6d77db9322fe3f607dd477"
server_key_iv: "e529ce8fed32c790888b4bac94525506"
server_key_version: "encryptionkey@0"
I20260928 11:01:01.436753 18043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 11:01:01.440114 18058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:01.441432 18043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:01:01.441608 18043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "40eba88f0d4f4604b64f9c9689c1e1fd"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "6a36b06abf6d77db9322fe3f607dd477"
server_key_iv: "e529ce8fed32c790888b4bac94525506"
server_key_version: "encryptionkey@0"
I20260928 11:01:01.441795 18043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:01.467669 18043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:01.472008 18043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:01.472270 18043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:01.473214 18043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:01.474581 18043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:01.474654 18043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:01.474743 18043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:01.474783 18043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:01.502952 18043 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:42873
I20260928 11:01:01.502992 18170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:42873 every 8 connection(s)
I20260928 11:01:01.504180 18043 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:01:01.508349 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 18043
I20260928 11:01:01.508556 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:01:01.509313 4081 external_mini_cluster.cc:1428] Setting key 401c9a4095475df1b908d4154a57fe5d
Sep 28 11:01:01 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:01.525434 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:01.506688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:56497 (local address 127.3.252.126:44131)
0928 11:01:01.507149 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:01.507154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:01.507998 (+ 844us) server_negotiation.cc:407] Connection header received
0928 11:01:01.510343 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:01.510348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:01.510416 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:01.510510 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:01.512272 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:01.513128 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:01.515194 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:01.515586 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:01.515683 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:01.519554 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:01.519559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:01.519563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:01.519596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:01.522445 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:01.523160 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:01.523165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:01.523168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:01.523301 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:01.523707 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:01.523712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:01.523716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:01.524046 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:01.524179 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:01.524855 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 11:01:01.525089 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":249,"threads_started":1}
I20260928 11:01:01.526312 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:01.507146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:56497)
0928 11:01:01.507827 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:01.507873 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:01.510111 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:01.510758 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:01.510768 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:01.511322 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:01.512075 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:01.512100 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:01.513273 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:01.513278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:01.515039 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:01.515053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:01.515805 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:01.515812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:01.516086 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:01.516788 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:01.516815 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:01.519325 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:01.522588 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:01.522595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:01.522610 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:01.523040 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:01.523403 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:01.523406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:01.523409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:01.523609 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:01.524148 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:01.524153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:01.524485 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 11:01:01.524739 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 11:01:01.525134 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":275,"threads_started":1}
I20260928 11:01:01.527855 18171 heartbeater.cc:347] Connected to a master server at 127.3.252.126:44131
I20260928 11:01:01.528192 18171 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:01.529026 18171 heartbeater.cc:511] Master 127.3.252.126:44131 requested a full tablet report, sending...
I20260928 11:01:01.531450 17981 ts_manager.cc:194] Registered new tserver with Master: 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873)
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
I20260928 11:01:01.533453 17981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:56497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 11:01:01.560770 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:44131
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:01.876145 18177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:01.876448 18177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:01.876581 18177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:01.883821 18177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:01.883942 18177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:01.884097 18177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:01:01.893901 18177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:01.895221 18177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:01.896577 18177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:01.909875 18182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:01.909976 18183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:01.909935 18185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:01.910392 18177 server_base.cc:1034] running on GCE node
I20260928 11:01:01.911212 18177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:01.912084 18177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:01.913275 18177 hybrid_clock.cc:648] HybridClock initialized: now 1790593261913218 us; error 43 us; skew 500 ppm
I20260928 11:01:01.913594 18177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:01 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:01.916491 18177 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:01:01.917872 18177 webserver.cc:460] Webserver started at http://127.3.252.66:39081/ using document root <none> and password file <none>
I20260928 11:01:01.918529 18177 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:01.918623 18177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:01.918857 18177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:01.921602 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "7d757dc84c784929a0721c26a21fff34"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "d3a6b65adda1ba50730a0c3424b57d5a"
server_key_iv: "9ff1aa9eaf6a55f8a4bd37baf1486630"
server_key_version: "encryptionkey@0"
I20260928 11:01:01.922304 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "7d757dc84c784929a0721c26a21fff34"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "d3a6b65adda1ba50730a0c3424b57d5a"
server_key_iv: "9ff1aa9eaf6a55f8a4bd37baf1486630"
server_key_version: "encryptionkey@0"
I20260928 11:01:01.927000 18177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 11:01:01.930369 18192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:01.931736 18177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:01:01.931887 18177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "7d757dc84c784929a0721c26a21fff34"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "d3a6b65adda1ba50730a0c3424b57d5a"
server_key_iv: "9ff1aa9eaf6a55f8a4bd37baf1486630"
server_key_version: "encryptionkey@0"
I20260928 11:01:01.932066 18177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:01.975032 18177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:01.978902 18177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:01.979167 18177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:01.980063 18177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:01.981384 18177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:01.981464 18177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:01.981572 18177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:01.981619 18177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:02.010210 18177 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:35199
I20260928 11:01:02.010258 18304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:35199 every 8 connection(s)
I20260928 11:01:02.011384 18177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:01:02.021646 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 18177
I20260928 11:01:02.021862 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:01:02.022321 4081 external_mini_cluster.cc:1428] Setting key f98c9c70f78b907a5920261e0e9f5770
Sep 28 11:01:02 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:02.030788 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.013845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:54315 (local address 127.3.252.126:44131)
0928 11:01:02.014046 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.014060 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.015172 (+ 1112us) server_negotiation.cc:407] Connection header received
0928 11:01:02.016885 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.016890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.016961 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.017053 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:02.018512 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.019235 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.020656 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.021039 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.021139 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.024689 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:02.024696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:02.024700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:02.024738 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:02.027710 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.028387 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.028392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.028396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.028562 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.028939 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.028944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.028947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.029268 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:02.029415 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.030234 (+ 819us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.030459 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 11:01:02.031805 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.014342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:54315)
0928 11:01:02.014999 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.015054 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:02.016610 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:02.017245 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:02.017252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:02.017776 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:02.018360 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.018378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.019348 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.019351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.020533 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.020544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.021202 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.021206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.021412 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.022080 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:02.022103 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:02.024451 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:02.027838 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:02.027845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:02.027858 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:02.028270 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:02.028658 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:02.028661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:02.028663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:02.028842 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:02.029405 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:02.029412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:02.029813 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 11:01:02.030114 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 11:01:02.030646 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":268,"threads_started":1}
I20260928 11:01:02.033186 18305 heartbeater.cc:347] Connected to a master server at 127.3.252.126:44131
I20260928 11:01:02.033506 18305 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:02.034245 18305 heartbeater.cc:511] Master 127.3.252.126:44131 requested a full tablet report, sending...
I20260928 11:01:02.035672 17981 ts_manager.cc:194] Registered new tserver with Master: 7d757dc84c784929a0721c26a21fff34 (127.3.252.66:35199)
I20260928 11:01:02.036540 17981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:54315
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 11:01:02.072997 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:44131
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:02.395960 18310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:02.396242 18310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:02.396378 18310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:02.405973 18310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:02.406109 18310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:02.406272 18310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:01:02.416206 18310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:02.417500 18310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:02.418910 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:02.431775 18315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:02.431823 18316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:02.432651 18310 server_base.cc:1034] running on GCE node
W20260928 11:01:02.431783 18318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:02.433543 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:02.434506 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:02.435729 18310 hybrid_clock.cc:648] HybridClock initialized: now 1790593262435677 us; error 44 us; skew 500 ppm
I20260928 11:01:02.436074 18310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:02 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:02.439199 18310 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:01:02.440647 18310 webserver.cc:460] Webserver started at http://127.3.252.67:43915/ using document root <none> and password file <none>
I20260928 11:01:02.441313 18310 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:02.441425 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:02.441708 18310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:02.444587 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "eea88a68ad7e4f4abad80402d7779332"
format_stamp: "Formatted at 2026-09-28 11:01:02 on dist-test-slave-djjw"
server_key: "398c230d3dffad9620fc4b5e40534fd2"
server_key_iv: "1b85b19d36c91c1a2e809a725426afb4"
server_key_version: "encryptionkey@0"
I20260928 11:01:02.445279 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "eea88a68ad7e4f4abad80402d7779332"
format_stamp: "Formatted at 2026-09-28 11:01:02 on dist-test-slave-djjw"
server_key: "398c230d3dffad9620fc4b5e40534fd2"
server_key_iv: "1b85b19d36c91c1a2e809a725426afb4"
server_key_version: "encryptionkey@0"
I20260928 11:01:02.450233 18310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 11:01:02.453491 18325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:02.454794 18310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:01:02.454998 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "eea88a68ad7e4f4abad80402d7779332"
format_stamp: "Formatted at 2026-09-28 11:01:02 on dist-test-slave-djjw"
server_key: "398c230d3dffad9620fc4b5e40534fd2"
server_key_iv: "1b85b19d36c91c1a2e809a725426afb4"
server_key_version: "encryptionkey@0"
I20260928 11:01:02.455231 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:02.495428 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:02.499322 18310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:02.499606 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:02.500533 18310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:02.501848 18310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:02.501926 18310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:02.502066 18310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:02.502120 18310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:02.531096 18310 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:35283
I20260928 11:01:02.531178 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:35283 every 8 connection(s)
I20260928 11:01:02.532341 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:01:02.534209 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 18310
I20260928 11:01:02.534420 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:01:02.534981 4081 external_mini_cluster.cc:1428] Setting key 13a6092717d587bc0ad661746a7965f8
I20260928 11:01:02.536638 18171 heartbeater.cc:503] Master 127.3.252.126:44131 was elected leader, sending a full tablet report...
Sep 28 11:01:02 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:02.552106 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.534862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:40461 (local address 127.3.252.126:44131)
0928 11:01:02.535376 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.535380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.537023 (+ 1643us) server_negotiation.cc:407] Connection header received
0928 11:01:02.539034 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.539038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.539098 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.539183 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:02.540594 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.541336 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.542853 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.543243 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.543346 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.546644 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:02.546649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:02.546652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:02.546685 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:02.549254 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.550008 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.550012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.550014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.550154 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.550608 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.550613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.550615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.550886 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:02.551001 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.551673 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.551841 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":286,"threads_started":1}
I20260928 11:01:02.552953 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.535535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:40461)
0928 11:01:02.536300 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.536897 (+ 597us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:02.538800 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:02.539365 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:02.539372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:02.539850 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:02.540430 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.540447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.541447 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.541450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.542703 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.542714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.543431 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.543436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.543633 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.544250 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:02.544271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:02.546447 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:02.549383 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:02.549402 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:02.549418 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:02.549888 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:02.550265 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:02.550269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:02.550271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:02.550496 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:02.550990 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:02.550996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:02.551302 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 11:01:02.551522 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 11:01:02.551902 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
I20260928 11:01:02.554230 18438 heartbeater.cc:347] Connected to a master server at 127.3.252.126:44131
I20260928 11:01:02.554508 18438 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:02.555187 18438 heartbeater.cc:511] Master 127.3.252.126:44131 requested a full tablet report, sending...
I20260928 11:01:02.556435 17981 ts_manager.cc:194] Registered new tserver with Master: eea88a68ad7e4f4abad80402d7779332 (127.3.252.67:35283)
I20260928 11:01:02.557194 17981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:40461
I20260928 11:01:02.562690 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:01:02.578536 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.566984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.3.252.126:44131)
0928 11:01:02.567129 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.567133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.567535 (+ 402us) server_negotiation.cc:407] Connection header received
0928 11:01:02.567668 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.567672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.567734 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.567812 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:02.569061 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.569792 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.571234 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.571672 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.571762 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.574044 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:02.574049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:02.574058 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:02.574089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:02.576455 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.577001 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.577004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.577007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.577121 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.577380 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.577383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.577384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.577702 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:02.577913 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.578133 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.578291 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 11:01:02.582211 17981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:01:02.588901 17981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:01:02.622774 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.600391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:35283 (local address 127.0.0.1:37658)
0928 11:01:02.606312 (+ 5921us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.606332 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:02.606455 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:02.616347 (+ 9892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:02.616352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:02.616377 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:02.616766 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.616777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.618453 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.618457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.620001 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.620013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.621687 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.621693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.621906 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.621976 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 11:01:02.622113 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 11:01:02.622305 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5780,"mutex_wait_us":37,"thread_start_us":5617,"threads_started":1}
I20260928 11:01:02.622928 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.600158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37270)
0928 11:01:02.602435 (+ 2277us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.602478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:02.602665 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:02.615074 (+ 12409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:02.615083 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:02.615116 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:02.615515 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.615529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.617600 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.617605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.620221 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.620236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.622069 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.622075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.622255 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.622287 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 11:01:02.622351 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 11:01:02.622481 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2007,"mutex_wait_us":136,"thread_start_us":1834,"threads_started":1}
I20260928 11:01:02.623785 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.600911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.3.252.67:35283)
0928 11:01:02.615630 (+ 14719us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.615637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.615667 (+ 30us) server_negotiation.cc:407] Connection header received
0928 11:01:02.615761 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.615768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.615969 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.616943 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:02.617177 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.618325 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.620159 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.621496 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.621636 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.621900 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.623291 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.623463 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14531,"thread_start_us":15195,"threads_started":1}
I20260928 11:01:02.623883 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.600920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.3.252.65:42873)
0928 11:01:02.614340 (+ 13420us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.614352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.614390 (+ 38us) server_negotiation.cc:407] Connection header received
0928 11:01:02.614513 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.614521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.614804 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.615051 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:02.615911 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.617468 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.620372 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.621742 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.623165 (+ 1423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.623382 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.623454 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.623619 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13214,"thread_start_us":13040,"threads_started":1}
I20260928 11:01:02.624358 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.606662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.3.252.66:35199)
0928 11:01:02.614710 (+ 8048us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.614718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.615226 (+ 508us) server_negotiation.cc:407] Connection header received
0928 11:01:02.615319 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.615325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.615481 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.615633 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:02.616198 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.617340 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.621933 (+ 4593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.623341 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.623474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.623646 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.623891 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.624059 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7877,"thread_start_us":7716,"threads_started":1}
I20260928 11:01:02.624707 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.607840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:35199 (local address 127.0.0.1:59598)
0928 11:01:02.614975 (+ 7135us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.614994 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:02.615101 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:02.615647 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:02.615651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:02.615674 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:02.616013 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.616027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.617464 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.617469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.620227 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.620236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.623558 (+ 3322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.623564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.623717 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.623748 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 11:01:02.624081 (+ 333us) client_negotiation.cc:240] Negotiation successful
0928 11:01:02.624211 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6987,"mutex_wait_us":159,"thread_start_us":6859,"threads_started":1}
I20260928 11:01:02.627271 18106 tablet_service.cc:1467] Processing CreateTablet for tablet 31192236b629413997047b82f90aea18 (DEFAULT_TABLE table=test-table [id=d4155e8482fb4098b8e6963b6e071681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:02.627310 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 31192236b629413997047b82f90aea18 (DEFAULT_TABLE table=test-table [id=d4155e8482fb4098b8e6963b6e071681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:02.628441 18106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31192236b629413997047b82f90aea18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:02.628695 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31192236b629413997047b82f90aea18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:02.629254 18240 tablet_service.cc:1467] Processing CreateTablet for tablet 31192236b629413997047b82f90aea18 (DEFAULT_TABLE table=test-table [id=d4155e8482fb4098b8e6963b6e071681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:02.630515 18240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31192236b629413997047b82f90aea18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:02.639834 18457 tablet_bootstrap.cc:492] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Bootstrap starting.
I20260928 11:01:02.642403 18458 tablet_bootstrap.cc:492] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Bootstrap starting.
I20260928 11:01:02.643265 18457 tablet_bootstrap.cc:654] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:02.643790 18459 tablet_bootstrap.cc:492] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Bootstrap starting.
I20260928 11:01:02.644639 18457 log.cc:824] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:02.645701 18458 tablet_bootstrap.cc:654] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:02.646735 18459 tablet_bootstrap.cc:654] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:02.646855 18458 log.cc:824] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:02.647748 18457 tablet_bootstrap.cc:492] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: No bootstrap required, opened a new log
I20260928 11:01:02.647737 18459 log.cc:824] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:02.648260 18457 ts_tablet_manager.cc:1397] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 11:01:02.649735 18458 tablet_bootstrap.cc:492] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: No bootstrap required, opened a new log
I20260928 11:01:02.650002 18458 ts_tablet_manager.cc:1397] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260928 11:01:02.651571 18459 tablet_bootstrap.cc:492] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: No bootstrap required, opened a new log
I20260928 11:01:02.651815 18459 ts_tablet_manager.cc:1397] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 11:01:02.655573 18457 raft_consensus.cc:348] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.655928 18457 raft_consensus.cc:374] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:02.656035 18457 raft_consensus.cc:729] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40eba88f0d4f4604b64f9c9689c1e1fd, State: Initialized, Role: FOLLOWER
I20260928 11:01:02.656596 18457 consensus_queue.cc:260] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.658293 18457 ts_tablet_manager.cc:1428] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 11:01:02.660269 18458 raft_consensus.cc:348] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.660701 18458 raft_consensus.cc:374] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:02.660792 18458 raft_consensus.cc:729] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea88a68ad7e4f4abad80402d7779332, State: Initialized, Role: FOLLOWER
I20260928 11:01:02.661397 18458 consensus_queue.cc:260] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.661821 18459 raft_consensus.cc:348] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.662313 18459 raft_consensus.cc:374] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:02.662401 18459 raft_consensus.cc:729] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d757dc84c784929a0721c26a21fff34, State: Initialized, Role: FOLLOWER
I20260928 11:01:02.663094 18459 consensus_queue.cc:260] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.664230 18305 heartbeater.cc:503] Master 127.3.252.126:44131 was elected leader, sending a full tablet report...
I20260928 11:01:02.664932 18464 raft_consensus.cc:482] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:01:02.665179 18464 raft_consensus.cc:504] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.665548 18438 heartbeater.cc:503] Master 127.3.252.126:44131 was elected leader, sending a full tablet report...
I20260928 11:01:02.665941 18458 ts_tablet_manager.cc:1428] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260928 11:01:02.667410 18459 ts_tablet_manager.cc:1428] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260928 11:01:02.668294 18464 leader_election.cc:288] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873), 7d757dc84c784929a0721c26a21fff34 (127.3.252.66:35199)
I20260928 11:01:02.673638 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.667937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:51357 (local address 127.3.252.65:42873)
0928 11:01:02.668131 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.668136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.668153 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:01:02.668216 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.668221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.668295 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.668393 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:02.668815 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.669889 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.672056 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.672859 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.672999 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.673072 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.673358 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.673453 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 11:01:02.673739 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.667385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.3.252.67:51357)
0928 11:01:02.667601 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.667626 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:02.667737 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:02.668433 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:02.668437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:02.668464 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:02.668658 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.668671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.670071 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.670077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.671910 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.671922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.672999 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.673005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.673218 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.673248 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 11:01:02.673338 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 11:01:02.673467 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":19}
I20260928 11:01:02.674757 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31192236b629413997047b82f90aea18" candidate_uuid: "eea88a68ad7e4f4abad80402d7779332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" is_pre_election: true
I20260928 11:01:02.675185 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.668038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:35199 (local address 127.3.252.67:38003)
0928 11:01:02.669278 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.669294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:02.669404 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:02.669817 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:02.669821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:02.669878 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:02.670320 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.670333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.671710 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.671716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.673167 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:02.673182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.674601 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.674606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:02.674718 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.674741 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 11:01:02.674810 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 11:01:02.674894 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":29,"thread_start_us":999,"threads_started":1}
I20260928 11:01:02.675205 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.668057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:38003 (local address 127.3.252.66:35199)
0928 11:01:02.668213 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.668217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.669381 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 11:01:02.669517 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.669522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.669587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.669667 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:02.670509 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.671567 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.673311 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.674421 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.674574 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.674642 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.674858 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.674985 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 11:01:02.675324 18126 raft_consensus.cc:2454] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea88a68ad7e4f4abad80402d7779332 in term 0.
I20260928 11:01:02.675908 18329 leader_election.cc:302] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40eba88f0d4f4604b64f9c9689c1e1fd, eea88a68ad7e4f4abad80402d7779332; no voters:
I20260928 11:01:02.676242 18464 raft_consensus.cc:2781] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:01:02.676349 18464 raft_consensus.cc:482] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:01:02.676234 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31192236b629413997047b82f90aea18" candidate_uuid: "eea88a68ad7e4f4abad80402d7779332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d757dc84c784929a0721c26a21fff34" is_pre_election: true
I20260928 11:01:02.676437 18464 raft_consensus.cc:3037] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:02.676658 18260 raft_consensus.cc:2454] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea88a68ad7e4f4abad80402d7779332 in term 0.
I20260928 11:01:02.679152 18464 raft_consensus.cc:504] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.679747 18464 leader_election.cc:288] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [CANDIDATE]: Term 1 election: Requested vote from peers 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873), 7d757dc84c784929a0721c26a21fff34 (127.3.252.66:35199)
I20260928 11:01:02.680087 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31192236b629413997047b82f90aea18" candidate_uuid: "eea88a68ad7e4f4abad80402d7779332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd"
I20260928 11:01:02.680157 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31192236b629413997047b82f90aea18" candidate_uuid: "eea88a68ad7e4f4abad80402d7779332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d757dc84c784929a0721c26a21fff34"
I20260928 11:01:02.680262 18126 raft_consensus.cc:3037] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:02.680326 18260 raft_consensus.cc:3037] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:02.682411 18126 raft_consensus.cc:2454] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea88a68ad7e4f4abad80402d7779332 in term 1.
I20260928 11:01:02.682487 18260 raft_consensus.cc:2454] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea88a68ad7e4f4abad80402d7779332 in term 1.
I20260928 11:01:02.682789 18329 leader_election.cc:302] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40eba88f0d4f4604b64f9c9689c1e1fd, eea88a68ad7e4f4abad80402d7779332; no voters:
I20260928 11:01:02.683043 18464 raft_consensus.cc:2781] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:01:02.683192 18464 raft_consensus.cc:686] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 1 LEADER]: Becoming Leader. State: Replica: eea88a68ad7e4f4abad80402d7779332, State: Running, Role: LEADER
I20260928 11:01:02.683616 18464 consensus_queue.cc:237] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:02.688113 17981 catalog_manager.cc:5495] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 reported cstate change: term changed from 0 to 1, leader changed from <none> to eea88a68ad7e4f4abad80402d7779332 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "eea88a68ad7e4f4abad80402d7779332" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } health_report { overall_health: UNKNOWN } } }
I20260928 11:01:02.709357 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.704213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.3.252.67:35283)
0928 11:01:02.704393 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.704398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.704416 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:01:02.704500 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.704504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.704585 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.704699 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:01:02.705003 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.705747 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.707201 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.707812 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.707936 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.708372 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:01:02.708881 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:01:02.709035 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.709093 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.709162 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 11:01:02.716997 18260 raft_consensus.cc:1261] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 1 FOLLOWER]: Refusing update from remote peer eea88a68ad7e4f4abad80402d7779332: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:02.717021 18126 raft_consensus.cc:1261] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Refusing update from remote peer eea88a68ad7e4f4abad80402d7779332: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:02.717933 18464 consensus_queue.cc:1035] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:02.718221 18466 consensus_queue.cc:1035] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:02.730612 18472 mvcc.cc:205] Tried to move back new op lower bound from 7334270004079976448 to 7334270003955572736. Current Snapshot: MvccSnapshot[applied={T|T < 7334270004079976448}]
I20260928 11:01:02.731612 18471 mvcc.cc:205] Tried to move back new op lower bound from 7334270004079976448 to 7334270003955572736. Current Snapshot: MvccSnapshot[applied={T|T < 7334270004079976448}]
I20260928 11:01:02.746863 17980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:02.747205 17980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:02.751101 17980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Sending DeleteTablet for 3 replicas of tablet 31192236b629413997047b82f90aea18
I20260928 11:01:02.752230 18106 tablet_service.cc:1514] Processing DeleteTablet for tablet 31192236b629413997047b82f90aea18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:02 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:37270
I20260928 11:01:02.752384 18240 tablet_service.cc:1514] Processing DeleteTablet for tablet 31192236b629413997047b82f90aea18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:02 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:59598
I20260928 11:01:02.752511 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 18043
I20260928 11:01:02.753726 18373 tablet_service.cc:1514] Processing DeleteTablet for tablet 31192236b629413997047b82f90aea18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:02 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:37658
I20260928 11:01:02.770361 18483 tablet_replica.cc:330] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: stopping tablet replica
I20260928 11:01:02.770895 18483 raft_consensus.cc:2229] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:01:02.771661 18481 tablet_replica.cc:330] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: stopping tablet replica
I20260928 11:01:02.771680 18483 raft_consensus.cc:2258] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:02.772357 18481 raft_consensus.cc:2229] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:02.772866 18481 raft_consensus.cc:2258] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:02.773599 18480 tablet_replica.cc:330] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: stopping tablet replica
I20260928 11:01:02.773758 18483 ts_tablet_manager.cc:1905] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:02.774068 18480 raft_consensus.cc:2229] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:02.774533 18480 raft_consensus.cc:2258] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:02.776316 18481 ts_tablet_manager.cc:1905] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 11:01:02 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 11:01:02.780932 18483 ts_tablet_manager.cc:1918] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:02.781121 18483 log.cc:1196] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/31192236b629413997047b82f90aea18
I20260928 11:01:02.781638 18483 ts_tablet_manager.cc:1939] T 31192236b629413997047b82f90aea18 P eea88a68ad7e4f4abad80402d7779332: Deleting consensus metadata
I20260928 11:01:02.783301 18481 ts_tablet_manager.cc:1918] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:02.783464 18481 log.cc:1196] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/31192236b629413997047b82f90aea18
I20260928 11:01:02.783509 17967 catalog_manager.cc:4841] TS eea88a68ad7e4f4abad80402d7779332 (127.3.252.67:35283): tablet 31192236b629413997047b82f90aea18 (table test-table [id=d4155e8482fb4098b8e6963b6e071681]) successfully deleted
I20260928 11:01:02.783993 18481 ts_tablet_manager.cc:1939] T 31192236b629413997047b82f90aea18 P 7d757dc84c784929a0721c26a21fff34: Deleting consensus metadata
I20260928 11:01:02.785291 18480 ts_tablet_manager.cc:1905] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:02.785737 17967 catalog_manager.cc:4841] TS 7d757dc84c784929a0721c26a21fff34 (127.3.252.66:35199): tablet 31192236b629413997047b82f90aea18 (table test-table [id=d4155e8482fb4098b8e6963b6e071681]) successfully deleted
I20260928 11:01:02.787750 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.753757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.3.252.65:42873)
0928 11:01:02.753972 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.753979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.766251 (+ 12272us) server_negotiation.cc:407] Connection header received
0928 11:01:02.766799 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.766806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.766903 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.767009 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:02.769194 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.770555 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.772943 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.773669 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.773792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.779847 (+ 6055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:02.779863 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:02.779881 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:02.779942 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:02.783876 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.784879 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.784889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.784894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.785068 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.785592 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.785598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.785602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.786161 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:02.786477 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.787107 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.787357 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 11:01:02.788519 18166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:02.877717 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 18043
I20260928 11:01:02.900372 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 18177
W20260928 11:01:02.900552 17968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.252.65:42873 (error 108)
I20260928 11:01:02.903290 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.902811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37280)
0928 11:01:02.902995 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.903113 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 11:01:02.903659 17968 catalog_manager.cc:4584] TS 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31192236b629413997047b82f90aea18: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Sep 28 11:01:02 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 11:01:02.916405 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.901209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.3.252.66:35199)
0928 11:01:02.901819 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:02.901847 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:02.901867 (+ 20us) server_negotiation.cc:407] Connection header received
0928 11:01:02.901945 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:02.901952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:02.902047 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:02.902137 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:02.903748 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.904861 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.906273 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:02.906700 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:02.906811 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:02.911039 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:02.911054 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:02.911068 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:02.911101 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:02.913793 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.914484 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.914491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.914493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.914610 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:02.914972 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:02.914976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:02.914977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:02.915339 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:02.915622 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:02.915888 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 11:01:02.916093 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461}
I20260928 11:01:02.917143 18300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:02.957239 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.956691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37296)
0928 11:01:02.956952 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.957058 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 11:01:02.958504 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:02.958048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37310)
0928 11:01:02.958259 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:02.958367 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 11:01:03.019284 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:03.018857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37318)
0928 11:01:03.019041 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:03.019137 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 11:01:03.019532 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 18177
I20260928 11:01:03.021385 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:03.020426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37328)
0928 11:01:03.021196 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:03.021266 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677}
I20260928 11:01:03.040963 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 18310
Sep 28 11:01:03 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 11:01:03.057632 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:03.041725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.3.252.67:35283)
0928 11:01:03.041925 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:03.041929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:03.042004 (+ 75us) server_negotiation.cc:407] Connection header received
0928 11:01:03.042191 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:03.042194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:03.042258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:03.042334 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:03.044156 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:03.044995 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:03.046664 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:03.047122 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:03.047233 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:03.051604 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:03.051618 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:03.051634 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:03.051677 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:03.054798 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:03.055523 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:03.055530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:03.055533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:03.055682 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:03.056064 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:03.056067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:03.056070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:03.056444 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:03.056744 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:03.057064 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 11:01:03.057281 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:01:03.058450 18433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:03.102187 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:03.101616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37340)
0928 11:01:03.101886 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:03.101991 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 11:01:03.103309 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:03.102841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37350)
0928 11:01:03.103105 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:03.103192 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 11:01:03.180599 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 18310
I20260928 11:01:03.196764 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 17949
I20260928 11:01:03.197929 18011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:03.308862 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 17949
I20260928 11:01:03.327186 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:44131
--webserver_interface=127.3.252.126
--webserver_port=34627
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:44131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:03.639909 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:03.640201 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:03.640303 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:03.647475 18489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:01:03.647590 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:03.647635 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:03.647696 18489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:01:03.647738 18489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:01:03.657501 18489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:44131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:03.658756 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:03.659970 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:03.681140 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:03.681195 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:03.681171 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:03.681879 18489 server_base.cc:1034] running on GCE node
I20260928 11:01:03.682772 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:03.684394 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:03.685602 18489 hybrid_clock.cc:648] HybridClock initialized: now 1790593263685533 us; error 65 us; skew 500 ppm
I20260928 11:01:03.686039 18489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:03 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593263, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:03.689349 18489 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:01:03.690987 18489 webserver.cc:460] Webserver started at http://127.3.252.126:34627/ using document root <none> and password file <none>
I20260928 11:01:03.691694 18489 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:03.691825 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:03.697773 18489 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 11:01:03.700590 18505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:03.702021 18489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:01:03.702178 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "7109a2c4793a4783bc3baafa4cfbc54e"
format_stamp: "Formatted at 2026-09-28 11:01:00 on dist-test-slave-djjw"
server_key: "cdf6f46a18553ed31cfbf92523f59f6e"
server_key_iv: "4280da3b5e2ae4396274dc5b13e8e8aa"
server_key_version: "encryptionkey@0"
I20260928 11:01:03.703190 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:03.725395 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:03.729444 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:03.729696 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:03.746973 18489 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:44131
I20260928 11:01:03.746990 18556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:44131 every 8 connection(s)
I20260928 11:01:03.748414 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:01:03.755045 18557 sys_catalog.cc:263] Verifying existing consensus state
I20260928 11:01:03.757694 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 18489
I20260928 11:01:03.758659 18557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Bootstrap starting.
I20260928 11:01:03.758742 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:42873
--local_ip_for_outbound_sockets=127.3.252.65
--tserver_master_addrs=127.3.252.126:44131
--webserver_port=36285
--webserver_interface=127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:01:03.780028 18557 log.cc:824] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:03.791965 18557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 11:01:03.792456 18557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Bootstrap complete.
I20260928 11:01:03.801257 18557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:03.802109 18557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7109a2c4793a4783bc3baafa4cfbc54e, State: Initialized, Role: FOLLOWER
I20260928 11:01:03.802664 18557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:03.802886 18557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:01:03.802986 18557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:01:03.803138 18557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 1 FOLLOWER]: Advancing to term 2
I20260928 11:01:03.806932 18557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:03.807425 18557 leader_election.cc:302] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7109a2c4793a4783bc3baafa4cfbc54e; no voters:
I20260928 11:01:03.808013 18557 leader_election.cc:288] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 11:01:03.808144 18561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 11:01:03.810266 18557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:01:03.814107 18561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 LEADER]: Becoming Leader. State: Replica: 7109a2c4793a4783bc3baafa4cfbc54e, State: Running, Role: LEADER
I20260928 11:01:03.814710 18561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:03.817446 18563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7109a2c4793a4783bc3baafa4cfbc54e. Latest consensus state: current_term: 2 leader_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } } }
I20260928 11:01:03.817951 18563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:03.817533 18562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } } }
I20260928 11:01:03.818692 18562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:03.822207 18570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:01:03.829785 18570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:01:03.831157 18570 catalog_manager.cc:1259] Loaded cluster ID: 03cfc9dd1c0545cb9535c0ea82c4b12c
I20260928 11:01:03.831259 18570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:01:03.834188 18570 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 11:01:03.850240 18570 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 11:01:03.859239 18570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:01:03.860901 18570 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Loaded TSK: 0
I20260928 11:01:03.862463 18570 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 11:01:04.093706 18559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:04.094038 18559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:04.094179 18559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:04.101498 18559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:04.101614 18559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:04.101759 18559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:01:04.112219 18559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:42873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=36285
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:04.113556 18559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:04.114907 18559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:04.134151 18584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:04.134193 18587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:04.134196 18585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:04.134807 18559 server_base.cc:1034] running on GCE node
I20260928 11:01:04.135509 18559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:04.136452 18559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:04.137647 18559 hybrid_clock.cc:648] HybridClock initialized: now 1790593264137589 us; error 51 us; skew 500 ppm
I20260928 11:01:04.137982 18559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:04 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:04.140929 18559 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:01:04.142455 18559 webserver.cc:460] Webserver started at http://127.3.252.65:36285/ using document root <none> and password file <none>
I20260928 11:01:04.143141 18559 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:04.143256 18559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:04.148587 18559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 11:01:04.151589 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:04.153045 18559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 11:01:04.153194 18559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "40eba88f0d4f4604b64f9c9689c1e1fd"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "6a36b06abf6d77db9322fe3f607dd477"
server_key_iv: "e529ce8fed32c790888b4bac94525506"
server_key_version: "encryptionkey@0"
I20260928 11:01:04.154142 18559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:04.183212 18559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:04.187306 18559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:04.187561 18559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:04.188495 18559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:04.190186 18601 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 11:01:04.195276 18601 ts_tablet_manager.cc:1852] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 11:01:04.196205 18601 ts_tablet_manager.cc:1905] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:04.203231 18601 ts_tablet_manager.cc:1918] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:04.203414 18601 log.cc:1196] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/31192236b629413997047b82f90aea18
I20260928 11:01:04.203843 18601 ts_tablet_manager.cc:1939] T 31192236b629413997047b82f90aea18 P 40eba88f0d4f4604b64f9c9689c1e1fd: Deleting consensus metadata
I20260928 11:01:04.204164 18559 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 11:01:04.204279 18559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260928 11:01:04.204392 18559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:04.204461 18559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:04.235072 18559 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:42873
I20260928 11:01:04.235145 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:42873 every 8 connection(s)
I20260928 11:01:04.236755 18559 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:01:04.244669 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 18559
I20260928 11:01:04.245477 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:35199
--local_ip_for_outbound_sockets=127.3.252.66
--tserver_master_addrs=127.3.252.126:44131
--webserver_port=39081
--webserver_interface=127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 11:01:04 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:04.263612 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:04.239961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:55345)
0928 11:01:04.240729 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:04.240766 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:04.242515 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:04.243936 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:04.243944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:04.244740 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:04.245569 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:04.245607 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.247398 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.247403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:04.249263 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:04.249282 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.251355 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.251362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:04.251638 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:04.252412 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:04.252437 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:04.254637 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:04.259048 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:04.259056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:04.259069 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:04.259576 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:04.260091 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:04.260095 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:04.260097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:04.260289 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:04.261149 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:04.261156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:04.261468 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 11:01:04.261725 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 11:01:04.262204 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":290,"threads_started":1}
I20260928 11:01:04.263727 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:04.239711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:55345 (local address 127.3.252.126:44131)
0928 11:01:04.240270 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:04.240286 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:04.240895 (+ 609us) server_negotiation.cc:407] Connection header received
0928 11:01:04.243070 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:04.243084 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:04.243436 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:04.243744 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:04.245901 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.247207 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.249572 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.251068 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.251237 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:04.255009 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:04.255022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:04.255045 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:04.255098 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:04.258882 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:04.259716 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:04.259725 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:04.259733 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:04.259954 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:04.260387 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:04.260392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:04.260394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:04.261029 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:04.261207 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:04.262015 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 11:01:04.262432 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":213,"threads_started":1}
I20260928 11:01:04.266883 18708 heartbeater.cc:347] Connected to a master server at 127.3.252.126:44131
I20260928 11:01:04.267207 18708 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:04.267920 18708 heartbeater.cc:511] Master 127.3.252.126:44131 requested a full tablet report, sending...
I20260928 11:01:04.270044 18522 ts_manager.cc:194] Registered new tserver with Master: 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873)
I20260928 11:01:04.271551 18522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:55345
W20260928 11:01:04.563776 18712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:04.564064 18712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:04.564190 18712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:04.571420 18712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:04.571560 18712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:04.571718 18712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:01:04.581383 18712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:35199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=39081
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:04.582718 18712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:04.584035 18712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:04.596460 18717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:04.596499 18718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:04.596519 18720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:04.597651 18712 server_base.cc:1034] running on GCE node
I20260928 11:01:04.598241 18712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:04.599159 18712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:04.600344 18712 hybrid_clock.cc:648] HybridClock initialized: now 1790593264600297 us; error 38 us; skew 500 ppm
I20260928 11:01:04.600639 18712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:04 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:04.603513 18712 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:01:04.604867 18712 webserver.cc:460] Webserver started at http://127.3.252.66:39081/ using document root <none> and password file <none>
I20260928 11:01:04.605540 18712 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:04.605662 18712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:04.611044 18712 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 11:01:04.613651 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:04.614897 18712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:01:04.615042 18712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "7d757dc84c784929a0721c26a21fff34"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "d3a6b65adda1ba50730a0c3424b57d5a"
server_key_iv: "9ff1aa9eaf6a55f8a4bd37baf1486630"
server_key_version: "encryptionkey@0"
I20260928 11:01:04.615963 18712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:04.655766 18712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:04.660038 18712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:04.660296 18712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:04.661280 18712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:04.662680 18712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:04.662776 18712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:04.662876 18712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:04.662941 18712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:04.691529 18712 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:35199
I20260928 11:01:04.691596 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:35199 every 8 connection(s)
I20260928 11:01:04.692927 18712 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:01:04.700647 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 18712
I20260928 11:01:04.701363 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:35283
--local_ip_for_outbound_sockets=127.3.252.67
--tserver_master_addrs=127.3.252.126:44131
--webserver_port=43915
--webserver_interface=127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 11:01:04 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:04.714520 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:04.695477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:52871 (local address 127.3.252.126:44131)
0928 11:01:04.695674 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:04.695679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:04.696774 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 11:01:04.698522 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:04.698526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:04.698593 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:04.698683 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:04.700073 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.700759 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.702978 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.703415 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.703529 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:04.707700 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:04.707707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:04.707711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:04.707752 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:04.710538 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:04.711286 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:04.711290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:04.711294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:04.711437 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:04.711962 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:04.711966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:04.711969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:04.712306 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:04.713958 (+ 1652us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:04.714095 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 11:01:04.714281 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 11:01:04.715099 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:04.695912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:52871)
0928 11:01:04.696617 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:04.696660 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:04.698283 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:04.698878 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:04.698886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:04.699355 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:04.699943 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:04.699960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.700914 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.700920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:04.702792 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:04.702806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:04.703634 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:04.703641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:04.703928 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:04.704694 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:04.704724 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:04.707442 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:04.710679 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:04.710703 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:04.710723 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:04.711169 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:04.711527 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:04.711530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:04.711532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:04.711843 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:04.712478 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:04.712487 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:04.712940 (+ 453us) client_negotiation.cc:769] Sending connection context
0928 11:01:04.713207 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 11:01:04.713717 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":260,"threads_started":1}
I20260928 11:01:04.716533 18840 heartbeater.cc:347] Connected to a master server at 127.3.252.126:44131
I20260928 11:01:04.716821 18840 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:04.717531 18840 heartbeater.cc:511] Master 127.3.252.126:44131 requested a full tablet report, sending...
I20260928 11:01:04.719090 18522 ts_manager.cc:194] Registered new tserver with Master: 7d757dc84c784929a0721c26a21fff34 (127.3.252.66:35199)
I20260928 11:01:04.720249 18522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:52871
W20260928 11:01:05.024956 18843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:05.025245 18843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:05.025339 18843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:05.032692 18843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:05.032814 18843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:05.032923 18843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:01:05.042860 18843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:35283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=43915
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:05.044143 18843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:05.045429 18843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:05.058192 18848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:05.058267 18849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:05.058224 18851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:05.059070 18843 server_base.cc:1034] running on GCE node
I20260928 11:01:05.059693 18843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:05.060645 18843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:05.061903 18843 hybrid_clock.cc:648] HybridClock initialized: now 1790593265061839 us; error 50 us; skew 500 ppm
I20260928 11:01:05.062258 18843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:05 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:05.065554 18843 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:01:05.067092 18843 webserver.cc:460] Webserver started at http://127.3.252.67:43915/ using document root <none> and password file <none>
I20260928 11:01:05.067792 18843 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:05.067924 18843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:05.073216 18843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260928 11:01:05.075887 18858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:05.077163 18843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 11:01:05.077318 18843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "eea88a68ad7e4f4abad80402d7779332"
format_stamp: "Formatted at 2026-09-28 11:01:02 on dist-test-slave-djjw"
server_key: "398c230d3dffad9620fc4b5e40534fd2"
server_key_iv: "1b85b19d36c91c1a2e809a725426afb4"
server_key_version: "encryptionkey@0"
I20260928 11:01:05.078400 18843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:05.092883 18843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:05.096786 18843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:05.097038 18843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:05.098063 18843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:05.099368 18843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:05.099474 18843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:05.099573 18843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:05.099640 18843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:05.129470 18843 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:35283
I20260928 11:01:05.129520 18970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:35283 every 8 connection(s)
I20260928 11:01:05.130887 18843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:01:05.136147 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 18843
Sep 28 11:01:05 dist-test-slave-djjw krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:05.151525 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.137008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.3.252.126:44131)
0928 11:01:05.138148 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.138154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.138171 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:01:05.138219 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.138224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.138288 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.138397 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:05.139752 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.140836 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.142252 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.142858 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.142947 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.145724 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:05.145731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:05.145736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:05.145776 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:05.148849 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.149470 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.149475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.149479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.149631 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.150065 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.150070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.150074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.150374 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:05.150591 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.150891 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.151123 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":1034,"thread_start_us":930,"threads_started":1}
I20260928 11:01:05.151837 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.133342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:34939 (local address 127.3.252.126:44131)
0928 11:01:05.133521 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.133527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.134755 (+ 1228us) server_negotiation.cc:407] Connection header received
0928 11:01:05.136572 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.136578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.136671 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.136763 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:05.138725 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.139431 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.141442 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.141898 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.141999 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.145816 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:05.145821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:05.145824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:05.145901 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:05.148776 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.149513 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.149516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.149519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.149642 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.150102 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.150105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.150107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.150373 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:05.150506 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.151350 (+ 844us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.151550 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":61}
I20260928 11:01:05.153666 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.133806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:34939)
0928 11:01:05.134573 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:05.134614 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:05.136298 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:05.137011 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:05.137020 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:05.137721 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:05.138569 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.138591 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.139582 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.139586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.141313 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.141326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.142125 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.142132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.142434 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.143186 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:05.143210 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:05.145588 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:05.148924 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:05.148935 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:05.148955 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:05.149412 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:05.149746 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:05.149751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:05.149755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:05.149991 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:05.150503 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:05.150511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:05.150920 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 11:01:05.151998 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0928 11:01:05.152494 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":296,"threads_started":1}
I20260928 11:01:05.154903 18971 heartbeater.cc:347] Connected to a master server at 127.3.252.126:44131
I20260928 11:01:05.155181 18971 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:05.155821 18971 heartbeater.cc:511] Master 127.3.252.126:44131 requested a full tablet report, sending...
I20260928 11:01:05.157119 18522 ts_manager.cc:194] Registered new tserver with Master: eea88a68ad7e4f4abad80402d7779332 (127.3.252.67:35283)
I20260928 11:01:05.158027 18522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:34939
I20260928 11:01:05.166147 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:01:05.172698 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.166976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.3.252.126:44131)
0928 11:01:05.167159 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.167163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.167600 (+ 437us) server_negotiation.cc:407] Connection header received
0928 11:01:05.167818 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.167822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.167986 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.168110 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:01:05.168471 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.169203 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.170642 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.171079 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.171180 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.171646 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:01:05.172164 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:01:05.172271 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.172427 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.172498 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 11:01:05.173638 18522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:01:05.180742 18522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:01:05.203938 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.192380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.0.0.1:37368)
0928 11:01:05.193561 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:05.193596 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:05.193752 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:05.195519 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:05.195526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:05.195564 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:05.195998 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.196010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.198254 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.198259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.200889 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.200905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.202839 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.202846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.203059 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.203124 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 11:01:05.203237 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 11:01:05.203446 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":1,"thread_start_us":2849,"threads_started":1}
I20260928 11:01:05.204382 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.193012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.3.252.67:35283)
0928 11:01:05.194457 (+ 1445us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.194467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.195309 (+ 842us) server_negotiation.cc:407] Connection header received
0928 11:01:05.195604 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.195612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.195839 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.195998 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:05.196917 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.198630 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.201330 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.202842 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.202947 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.203205 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.203874 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.204045 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":1084,"threads_started":1}
I20260928 11:01:05.205157 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.192791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:35283 (local address 127.0.0.1:37676)
0928 11:01:05.195192 (+ 2401us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:05.195213 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:05.195420 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:05.196305 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:05.196311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:05.196337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:05.196725 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.196736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.198841 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.198847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.201136 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.201153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.203421 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.203427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.203655 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.203690 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 11:01:05.204654 (+ 964us) client_negotiation.cc:240] Negotiation successful
0928 11:01:05.204808 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2284,"thread_start_us":430,"threads_started":1}
I20260928 11:01:05.205274 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.192615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.3.252.65:42873)
0928 11:01:05.194868 (+ 2253us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.194876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.194904 (+ 28us) server_negotiation.cc:407] Connection header received
0928 11:01:05.195000 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.195007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.195284 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.196220 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:05.196432 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.198097 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.201127 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.202607 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.204449 (+ 1842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.204664 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.204731 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.204899 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2106,"thread_start_us":1984,"threads_started":1}
I20260928 11:01:05.207576 18643 tablet_service.cc:1467] Processing CreateTablet for tablet 8ceea19d803e4bea8c456d602221cfec (DEFAULT_TABLE table=test-table [id=91a4c2fa8fd44800bd39be5b5d9de8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:05.208928 18643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ceea19d803e4bea8c456d602221cfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:05.210026 18906 tablet_service.cc:1467] Processing CreateTablet for tablet 8ceea19d803e4bea8c456d602221cfec (DEFAULT_TABLE table=test-table [id=91a4c2fa8fd44800bd39be5b5d9de8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:05.211324 18906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ceea19d803e4bea8c456d602221cfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:05.213163 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.194785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:35199 (local address 127.0.0.1:59628)
0928 11:01:05.204178 (+ 9393us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:05.204196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:05.204281 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:05.206534 (+ 2253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:05.206538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:05.206562 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:05.206763 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.206773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.208727 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.208734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.210481 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.210497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.212402 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.212408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.212584 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.212618 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 11:01:05.212728 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 11:01:05.212867 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9309,"thread_start_us":11710,"threads_started":1}
I20260928 11:01:05.213397 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.195103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.3.252.66:35199)
0928 11:01:05.205906 (+ 10803us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.205915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.205948 (+ 33us) server_negotiation.cc:407] Connection header received
0928 11:01:05.206047 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.206055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.206325 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.206937 (+ 612us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:05.207216 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.208596 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.210730 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.212198 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.212329 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.212548 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.212778 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.212945 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10624,"thread_start_us":10511,"threads_started":1}
I20260928 11:01:05.217402 18775 tablet_service.cc:1467] Processing CreateTablet for tablet 8ceea19d803e4bea8c456d602221cfec (DEFAULT_TABLE table=test-table [id=91a4c2fa8fd44800bd39be5b5d9de8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:05.219189 18987 tablet_bootstrap.cc:492] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Bootstrap starting.
I20260928 11:01:05.219625 18775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ceea19d803e4bea8c456d602221cfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:05.222895 18987 tablet_bootstrap.cc:654] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:05.224136 18987 log.cc:824] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:05.225670 18989 tablet_bootstrap.cc:492] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Bootstrap starting.
I20260928 11:01:05.227396 18987 tablet_bootstrap.cc:492] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: No bootstrap required, opened a new log
I20260928 11:01:05.227651 18987 ts_tablet_manager.cc:1397] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 11:01:05.228519 18989 tablet_bootstrap.cc:654] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:05.229482 18989 log.cc:824] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:05.231988 18989 tablet_bootstrap.cc:492] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: No bootstrap required, opened a new log
I20260928 11:01:05.232240 18989 ts_tablet_manager.cc:1397] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 11:01:05.236019 18991 tablet_bootstrap.cc:492] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Bootstrap starting.
I20260928 11:01:05.240468 18991 tablet_bootstrap.cc:654] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:05.241863 18991 log.cc:824] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:05.241945 18989 raft_consensus.cc:348] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.242403 18989 raft_consensus.cc:374] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:05.242503 18989 raft_consensus.cc:729] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea88a68ad7e4f4abad80402d7779332, State: Initialized, Role: FOLLOWER
I20260928 11:01:05.243218 18989 consensus_queue.cc:260] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.243753 18987 raft_consensus.cc:348] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.244156 18987 raft_consensus.cc:374] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:05.244268 18987 raft_consensus.cc:729] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40eba88f0d4f4604b64f9c9689c1e1fd, State: Initialized, Role: FOLLOWER
I20260928 11:01:05.246330 18987 consensus_queue.cc:260] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.250877 18971 heartbeater.cc:503] Master 127.3.252.126:44131 was elected leader, sending a full tablet report...
I20260928 11:01:05.251506 18989 ts_tablet_manager.cc:1428] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20260928 11:01:05.252640 18991 tablet_bootstrap.cc:492] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: No bootstrap required, opened a new log
I20260928 11:01:05.252974 18991 ts_tablet_manager.cc:1397] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260928 11:01:05.256632 18987 ts_tablet_manager.cc:1428] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Time spent starting tablet: real 0.029s user 0.018s sys 0.004s
I20260928 11:01:05.256748 18708 heartbeater.cc:503] Master 127.3.252.126:44131 was elected leader, sending a full tablet report...
I20260928 11:01:05.260957 18991 raft_consensus.cc:348] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.261314 18991 raft_consensus.cc:374] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:05.261404 18991 raft_consensus.cc:729] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d757dc84c784929a0721c26a21fff34, State: Initialized, Role: FOLLOWER
I20260928 11:01:05.261972 18991 consensus_queue.cc:260] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.262913 18840 heartbeater.cc:503] Master 127.3.252.126:44131 was elected leader, sending a full tablet report...
I20260928 11:01:05.263367 18991 ts_tablet_manager.cc:1428] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
W20260928 11:01:05.382540 18972 tablet.cc:2369] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 11:01:05.444896 18841 tablet.cc:2369] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 11:01:05.489265 18709 tablet.cc:2369] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:01:05.555996 18995 raft_consensus.cc:482] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:01:05.556314 18995 raft_consensus.cc:504] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.558174 18995 leader_election.cc:288] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873), eea88a68ad7e4f4abad80402d7779332 (127.3.252.67:35283)
I20260928 11:01:05.566879 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.558493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:44827 (local address 127.3.252.65:42873)
0928 11:01:05.558771 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.558777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.559485 (+ 708us) server_negotiation.cc:407] Connection header received
0928 11:01:05.559667 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.559674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.559748 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.559839 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:05.560482 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.561593 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.564497 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.565748 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.565937 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.566012 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.566473 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.566631 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 11:01:05.566879 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.558602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:56791 (local address 127.3.252.67:35283)
0928 11:01:05.558801 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.558807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.559501 (+ 694us) server_negotiation.cc:407] Connection header received
0928 11:01:05.559777 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.559782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.559855 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.559931 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:05.560645 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.561662 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.564482 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.565752 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.565939 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.566012 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.566473 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.566603 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 11:01:05.567113 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.558667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:35283 (local address 127.3.252.66:56791)
0928 11:01:05.559392 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:05.559406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:05.559579 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:05.560027 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:05.560032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:05.560074 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:05.560485 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.560498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.561772 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.561780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.564314 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.564333 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.565997 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.566008 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.566306 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.566351 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 11:01:05.566448 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 11:01:05.566612 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":215,"spinlock_wait_cycles":25216,"thread_start_us":435,"threads_started":1}
I20260928 11:01:05.567134 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.558537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:42873 (local address 127.3.252.66:44827)
0928 11:01:05.559364 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:05.559387 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:05.559531 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:05.559867 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:05.559872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:05.559896 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:05.560334 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.560348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.561724 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.561729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.564339 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:05.564348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.566018 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.566023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:05.566301 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.566360 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 11:01:05.566447 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 11:01:05.566612 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":239,"spinlock_wait_cycles":39680,"thread_start_us":440,"threads_started":1}
I20260928 11:01:05.568094 18926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceea19d803e4bea8c456d602221cfec" candidate_uuid: "7d757dc84c784929a0721c26a21fff34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea88a68ad7e4f4abad80402d7779332" is_pre_election: true
I20260928 11:01:05.568223 18663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceea19d803e4bea8c456d602221cfec" candidate_uuid: "7d757dc84c784929a0721c26a21fff34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" is_pre_election: true
I20260928 11:01:05.568583 18926 raft_consensus.cc:2454] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d757dc84c784929a0721c26a21fff34 in term 0.
I20260928 11:01:05.568679 18663 raft_consensus.cc:2454] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d757dc84c784929a0721c26a21fff34 in term 0.
I20260928 11:01:05.569255 18730 leader_election.cc:302] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d757dc84c784929a0721c26a21fff34, eea88a68ad7e4f4abad80402d7779332; no voters:
I20260928 11:01:05.569643 18995 raft_consensus.cc:2781] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:01:05.569768 18995 raft_consensus.cc:482] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:01:05.569886 18995 raft_consensus.cc:3037] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:05.571949 18995 raft_consensus.cc:504] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.572558 18995 leader_election.cc:288] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [CANDIDATE]: Term 1 election: Requested vote from peers 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873), eea88a68ad7e4f4abad80402d7779332 (127.3.252.67:35283)
I20260928 11:01:05.572942 18663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceea19d803e4bea8c456d602221cfec" candidate_uuid: "7d757dc84c784929a0721c26a21fff34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd"
I20260928 11:01:05.573032 18926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceea19d803e4bea8c456d602221cfec" candidate_uuid: "7d757dc84c784929a0721c26a21fff34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea88a68ad7e4f4abad80402d7779332"
I20260928 11:01:05.573151 18663 raft_consensus.cc:3037] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:05.573221 18926 raft_consensus.cc:3037] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:05.576273 18663 raft_consensus.cc:2454] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d757dc84c784929a0721c26a21fff34 in term 1.
I20260928 11:01:05.576273 18926 raft_consensus.cc:2454] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d757dc84c784929a0721c26a21fff34 in term 1.
I20260928 11:01:05.576670 18730 leader_election.cc:302] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 40eba88f0d4f4604b64f9c9689c1e1fd, 7d757dc84c784929a0721c26a21fff34, eea88a68ad7e4f4abad80402d7779332; no voters:
I20260928 11:01:05.576999 18995 raft_consensus.cc:2781] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:01:05.577397 18995 raft_consensus.cc:686] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 1 LEADER]: Becoming Leader. State: Replica: 7d757dc84c784929a0721c26a21fff34, State: Running, Role: LEADER
I20260928 11:01:05.577874 18995 consensus_queue.cc:237] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } }
I20260928 11:01:05.582113 18522 catalog_manager.cc:5495] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d757dc84c784929a0721c26a21fff34 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "7d757dc84c784929a0721c26a21fff34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d757dc84c784929a0721c26a21fff34" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 35199 } health_report { overall_health: HEALTHY } } }
I20260928 11:01:05.660943 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.655778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.3.252.66:35199)
0928 11:01:05.655957 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.655961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.655978 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:01:05.656085 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.656088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.656163 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.656285 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:01:05.656657 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.657361 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.658967 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.659385 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.659481 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.659985 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:01:05.660478 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:01:05.660592 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.660628 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.660694 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 11:01:05.668797 18926 raft_consensus.cc:1261] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 1 FOLLOWER]: Refusing update from remote peer 7d757dc84c784929a0721c26a21fff34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:05.668809 18663 raft_consensus.cc:1261] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Refusing update from remote peer 7d757dc84c784929a0721c26a21fff34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:05.669628 18998 consensus_queue.cc:1035] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea88a68ad7e4f4abad80402d7779332" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:05.670037 18995 consensus_queue.cc:1035] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40eba88f0d4f4604b64f9c9689c1e1fd" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 42873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:05.680447 19003 mvcc.cc:205] Tried to move back new op lower bound from 7334270016170299392 to 7334270015810441216. Current Snapshot: MvccSnapshot[applied={T|T < 7334270016170299392}]
I20260928 11:01:05.680754 19001 mvcc.cc:205] Tried to move back new op lower bound from 7334270016170299392 to 7334270015810441216. Current Snapshot: MvccSnapshot[applied={T|T < 7334270016170299392}]
I20260928 11:01:05.684077 19002 mvcc.cc:205] Tried to move back new op lower bound from 7334270016170299392 to 7334270015810441216. Current Snapshot: MvccSnapshot[applied={T|T < 7334270016170299392}]
I20260928 11:01:05.694186 18522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:05.694420 18522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:05.697620 18522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Sending DeleteTablet for 3 replicas of tablet 8ceea19d803e4bea8c456d602221cfec
I20260928 11:01:05.698952 18775 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ceea19d803e4bea8c456d602221cfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:05 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:59628
I20260928 11:01:05.699621 18643 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ceea19d803e4bea8c456d602221cfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:05 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:37368
I20260928 11:01:05.699898 19010 tablet_replica.cc:330] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: stopping tablet replica
I20260928 11:01:05.700376 19010 raft_consensus.cc:2229] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:01:05.700543 18906 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ceea19d803e4bea8c456d602221cfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:05 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:37676
I20260928 11:01:05.701187 19010 raft_consensus.cc:2258] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:05.702059 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 18559
I20260928 11:01:05.709854 19010 ts_tablet_manager.cc:1905] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:05.724678 19010 ts_tablet_manager.cc:1918] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:05.724877 19010 log.cc:1196] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/8ceea19d803e4bea8c456d602221cfec
I20260928 11:01:05.725472 19010 ts_tablet_manager.cc:1939] T 8ceea19d803e4bea8c456d602221cfec P 7d757dc84c784929a0721c26a21fff34: Deleting consensus metadata
I20260928 11:01:05.725950 19012 tablet_replica.cc:330] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: stopping tablet replica
I20260928 11:01:05.726438 19012 raft_consensus.cc:2229] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:05.726967 19012 raft_consensus.cc:2258] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:05.727523 18508 catalog_manager.cc:4841] TS 7d757dc84c784929a0721c26a21fff34 (127.3.252.66:35199): tablet 8ceea19d803e4bea8c456d602221cfec (table test-table [id=91a4c2fa8fd44800bd39be5b5d9de8a3]) successfully deleted
I20260928 11:01:05.728343 19011 tablet_replica.cc:330] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: stopping tablet replica
I20260928 11:01:05.728794 19011 raft_consensus.cc:2229] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:05.729295 19011 raft_consensus.cc:2258] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:05.731729 19012 ts_tablet_manager.cc:1905] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:05.733132 19011 ts_tablet_manager.cc:1905] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:05.740164 19011 ts_tablet_manager.cc:1918] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:05.740415 19011 log.cc:1196] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/8ceea19d803e4bea8c456d602221cfec
I20260928 11:01:05.740595 19012 ts_tablet_manager.cc:1918] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:05.740787 19012 log.cc:1196] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/8ceea19d803e4bea8c456d602221cfec
I20260928 11:01:05.741063 19011 ts_tablet_manager.cc:1939] T 8ceea19d803e4bea8c456d602221cfec P 40eba88f0d4f4604b64f9c9689c1e1fd: Deleting consensus metadata
I20260928 11:01:05.741328 19012 ts_tablet_manager.cc:1939] T 8ceea19d803e4bea8c456d602221cfec P eea88a68ad7e4f4abad80402d7779332: Deleting consensus metadata
I20260928 11:01:05.743216 18509 catalog_manager.cc:4841] TS 40eba88f0d4f4604b64f9c9689c1e1fd (127.3.252.65:42873): tablet 8ceea19d803e4bea8c456d602221cfec (table test-table [id=91a4c2fa8fd44800bd39be5b5d9de8a3]) successfully deleted
I20260928 11:01:05.743793 18508 catalog_manager.cc:4841] TS eea88a68ad7e4f4abad80402d7779332 (127.3.252.67:35283): tablet 8ceea19d803e4bea8c456d602221cfec (table test-table [id=91a4c2fa8fd44800bd39be5b5d9de8a3]) successfully deleted
I20260928 11:01:05.756645 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.702893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.3.252.65:42873)
0928 11:01:05.703086 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.703094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.743478 (+ 40384us) server_negotiation.cc:407] Connection header received
0928 11:01:05.743732 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.743735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.743820 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.743905 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:05.745470 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.746228 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.747607 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.748023 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.748131 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.750851 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:05.750867 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:05.750883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:05.750919 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:05.753820 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.754595 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.754604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.754607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.754746 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.755112 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.755116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.755118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.755477 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:05.755762 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.756084 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.756295 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 11:01:05.757324 18703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:05.847258 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 18559
I20260928 11:01:05.862576 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 18712
I20260928 11:01:05.878294 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:05.863547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.3.252.66:35199)
0928 11:01:05.863764 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:05.863771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:05.863792 (+ 21us) server_negotiation.cc:407] Connection header received
0928 11:01:05.863858 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:05.863864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:05.863944 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:05.864056 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:05.865450 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.866634 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.868006 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:05.868681 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:05.868805 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:05.871691 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:05.871708 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:05.871727 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:05.871771 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:05.875503 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.876182 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.876193 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.876196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.876349 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:05.876704 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:05.876709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:05.876713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:05.877168 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:05.877483 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:05.877652 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 11:01:05.877902 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 11:01:05.879034 18835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:05.991318 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 18712
I20260928 11:01:06.007297 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 18843
I20260928 11:01:06.027570 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:06.008159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.3.252.67:35283)
0928 11:01:06.008335 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:06.008340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:06.008358 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:01:06.008477 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:06.008481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:06.008543 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:06.008646 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:06.009938 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:06.010728 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:06.012057 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:06.012442 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:06.012541 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:06.022259 (+ 9718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:06.022276 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:06.022293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:06.022336 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:06.024929 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:06.025696 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:06.025704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:06.025707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:06.025885 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:06.026240 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:06.026244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:06.026246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:06.026610 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:06.026830 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:06.027060 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 11:01:06.027234 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 11:01:06.028213 18966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:06.115231 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 18843
I20260928 11:01:06.130961 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 18489
I20260928 11:01:06.132064 18552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:06.243923 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 18489
I20260928 11:01:06.262360 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:44131
--webserver_interface=127.3.252.126
--webserver_port=34627
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:44131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:06.597470 19018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:06.597795 19018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:06.597939 19018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:06.605319 19018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:01:06.605441 19018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:06.605511 19018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:06.605592 19018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:01:06.605677 19018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:01:06.615705 19018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:44131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:06.617038 19018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:06.618345 19018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:06.629639 19026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:06.629669 19023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:06.629698 19024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:06.630432 19018 server_base.cc:1034] running on GCE node
I20260928 11:01:06.631268 19018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:06.632747 19018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:06.633927 19018 hybrid_clock.cc:648] HybridClock initialized: now 1790593266633875 us; error 39 us; skew 500 ppm
I20260928 11:01:06.634260 19018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:06 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:06.637318 19018 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:01:06.638847 19018 webserver.cc:460] Webserver started at http://127.3.252.126:34627/ using document root <none> and password file <none>
I20260928 11:01:06.639523 19018 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:06.639642 19018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:06.645292 19018 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 11:01:06.648056 19033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:06.649340 19018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 11:01:06.649489 19018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "7109a2c4793a4783bc3baafa4cfbc54e"
format_stamp: "Formatted at 2026-09-28 11:01:00 on dist-test-slave-djjw"
server_key: "cdf6f46a18553ed31cfbf92523f59f6e"
server_key_iv: "4280da3b5e2ae4396274dc5b13e8e8aa"
server_key_version: "encryptionkey@0"
I20260928 11:01:06.650477 19018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:06.666864 19018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:06.670756 19018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:06.671003 19018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:06.688021 19018 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:44131
I20260928 11:01:06.688028 19084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:44131 every 8 connection(s)
I20260928 11:01:06.689463 19018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:01:06.695802 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19018
I20260928 11:01:06.696322 19085 sys_catalog.cc:263] Verifying existing consensus state
I20260928 11:01:06.696745 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:42873
--local_ip_for_outbound_sockets=127.3.252.65
--tserver_master_addrs=127.3.252.126:44131
--webserver_port=36285
--webserver_interface=127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:01:06.700587 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Bootstrap starting.
I20260928 11:01:06.721342 19085 log.cc:824] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:06.737717 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 11:01:06.738234 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e: Bootstrap complete.
I20260928 11:01:06.747087 19085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:06.747946 19085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7109a2c4793a4783bc3baafa4cfbc54e, State: Initialized, Role: FOLLOWER
I20260928 11:01:06.748523 19085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:06.748733 19085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:01:06.748842 19085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:01:06.749006 19085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 2 FOLLOWER]: Advancing to term 3
I20260928 11:01:06.753067 19085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:06.753574 19085 leader_election.cc:302] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7109a2c4793a4783bc3baafa4cfbc54e; no voters:
I20260928 11:01:06.754259 19085 leader_election.cc:288] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 11:01:06.754400 19089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 11:01:06.756311 19085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:01:06.759889 19089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [term 3 LEADER]: Becoming Leader. State: Replica: 7109a2c4793a4783bc3baafa4cfbc54e, State: Running, Role: LEADER
I20260928 11:01:06.760562 19089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } }
I20260928 11:01:06.763250 19090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } } }
I20260928 11:01:06.763644 19090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:06.766650 19091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7109a2c4793a4783bc3baafa4cfbc54e. Latest consensus state: current_term: 3 leader_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7109a2c4793a4783bc3baafa4cfbc54e" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 44131 } } }
I20260928 11:01:06.766858 19091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7109a2c4793a4783bc3baafa4cfbc54e [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:06.767177 19099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:01:06.775610 19099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:01:06.777032 19099 catalog_manager.cc:1259] Loaded cluster ID: 03cfc9dd1c0545cb9535c0ea82c4b12c
I20260928 11:01:06.777129 19099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 11:01:06.779029 19099 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f615d9a09ad google::LogMessage::Fail() at ??:0
@ 0x7f615d9a388a google::LogMessage::SendToLog() at ??:0
@ 0x7f615d9a04de google::LogMessage::Flush() at ??:0
@ 0x7f615d9a1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f61733d3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f61733d0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6173458610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6173458439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f61605ec345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f61605ef91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f61605ef799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f61605caeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61638a46db start_thread at ??:0
@ 0x7f615c83a61f clone at ??:0
W20260928 11:01:07.081018 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:07.081284 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:07.081378 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:07.088563 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:07.088680 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:07.088801 19087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:01:07.098450 19087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:42873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=36285
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:07.099637 19087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:07.100879 19087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:07.113505 19128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:07.113555 19127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:07.113595 19130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:07.114459 19087 server_base.cc:1034] running on GCE node
I20260928 11:01:07.115064 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:07.115962 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:07.117165 19087 hybrid_clock.cc:648] HybridClock initialized: now 1790593267117119 us; error 29 us; skew 500 ppm
I20260928 11:01:07.117447 19087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:07 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593267, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:07.120424 19087 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:01:07.121747 19087 webserver.cc:460] Webserver started at http://127.3.252.65:36285/ using document root <none> and password file <none>
I20260928 11:01:07.122406 19087 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:07.122524 19087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:07.127516 19087 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 11:01:07.131829 19137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:07.133095 19087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:01:07.133235 19087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "40eba88f0d4f4604b64f9c9689c1e1fd"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "6a36b06abf6d77db9322fe3f607dd477"
server_key_iv: "e529ce8fed32c790888b4bac94525506"
server_key_version: "encryptionkey@0"
I20260928 11:01:07.134232 19087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:07.159686 19087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:07.163919 19087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:07.164180 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:07.165246 19087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:07.166623 19087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:07.166728 19087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:07.166844 19087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:07.166913 19087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:07.196556 19087 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:42873
I20260928 11:01:07.196636 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:42873 every 8 connection(s)
I20260928 11:01:07.197988 19087 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:01:07.201932 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19087
I20260928 11:01:07.202523 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:35199
--local_ip_for_outbound_sockets=127.3.252.66
--tserver_master_addrs=127.3.252.126:44131
--webserver_port=39081
--webserver_interface=127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:01:07.202589 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:07.200726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:57247)
0928 11:01:07.201419 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:07.201664 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":268,"threads_started":1}
W20260928 11:01:07.203214 19250 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:44131 (0 consecutive failures): Network error: Failed to ping master at 127.3.252.126:44131: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
I20260928 11:01:07.203991 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:07.203693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:36421)
0928 11:01:07.203828 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:07.203892 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 11:01:07.204732 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:07.204435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:54669)
0928 11:01:07.204568 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:07.204635 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 11:01:07.204928 19250 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 11:01:07.519870 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:07.520146 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:07.520262 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:07.527315 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:07.527429 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:07.527586 19253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:01:07.536931 19253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:35199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=39081
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:07.538247 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:07.539556 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:07.551877 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:07.551836 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:07.551877 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:07.552778 19253 server_base.cc:1034] running on GCE node
I20260928 11:01:07.553375 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:07.554382 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:07.555591 19253 hybrid_clock.cc:648] HybridClock initialized: now 1790593267555537 us; error 45 us; skew 500 ppm
I20260928 11:01:07.555887 19253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:07 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593267, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:07.558840 19253 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:01:07.560210 19253 webserver.cc:460] Webserver started at http://127.3.252.66:39081/ using document root <none> and password file <none>
I20260928 11:01:07.560864 19253 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:07.560976 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:07.566260 19253 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:01:07.568782 19268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:07.570237 19253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:01:07.570396 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "7d757dc84c784929a0721c26a21fff34"
format_stamp: "Formatted at 2026-09-28 11:01:01 on dist-test-slave-djjw"
server_key: "d3a6b65adda1ba50730a0c3424b57d5a"
server_key_iv: "9ff1aa9eaf6a55f8a4bd37baf1486630"
server_key_version: "encryptionkey@0"
I20260928 11:01:07.571403 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:07.596313 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:07.600312 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:07.600570 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:07.601518 19253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:07.602865 19253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:07.602957 19253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:07.603065 19253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:07.603133 19253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:07.632536 19253 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:35199
I20260928 11:01:07.632601 19380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:35199 every 8 connection(s)
I20260928 11:01:07.633953 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:01:07.638649 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:07.636729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:60151)
0928 11:01:07.637422 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:07.637679 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":256,"threads_started":1}
W20260928 11:01:07.639287 19381 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:44131 (0 consecutive failures): Network error: Failed to ping master at 127.3.252.126:44131: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
I20260928 11:01:07.640014 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:07.639721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:40971)
0928 11:01:07.639846 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:07.639913 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 11:01:07.640868 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:07.640581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:55107)
0928 11:01:07.640698 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:07.640769 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 11:01:07.641117 19381 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 11:01:07.643731 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19253
I20260928 11:01:07.644308 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:35283
--local_ip_for_outbound_sockets=127.3.252.67
--tserver_master_addrs=127.3.252.126:44131
--webserver_port=43915
--webserver_interface=127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:07.953334 19384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:07.953634 19384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:07.953773 19384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:07.961043 19384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:07.961180 19384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:07.961338 19384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:01:07.973621 19384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:35283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=43915
--tserver_master_addrs=127.3.252.126:44131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:07.975196 19384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:07.976640 19384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:07.992272 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:07.992230 19390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:07.992452 19392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:07.992885 19384 server_base.cc:1034] running on GCE node
I20260928 11:01:07.993541 19384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:07.994645 19384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:07.995893 19384 hybrid_clock.cc:648] HybridClock initialized: now 1790593267995837 us; error 41 us; skew 500 ppm
I20260928 11:01:07.996232 19384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:07 dist-test-slave-djjw krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593267, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:07.999297 19384 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:01:08.000808 19384 webserver.cc:460] Webserver started at http://127.3.252.67:43915/ using document root <none> and password file <none>
I20260928 11:01:08.001471 19384 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:08.001583 19384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:08.007011 19384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 11:01:08.009656 19399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:08.011044 19384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:01:08.011196 19384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "eea88a68ad7e4f4abad80402d7779332"
format_stamp: "Formatted at 2026-09-28 11:01:02 on dist-test-slave-djjw"
server_key: "398c230d3dffad9620fc4b5e40534fd2"
server_key_iv: "1b85b19d36c91c1a2e809a725426afb4"
server_key_version: "encryptionkey@0"
I20260928 11:01:08.012166 19384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:08.033576 19384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:08.037431 19384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:08.037675 19384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:08.038669 19384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:08.039928 19384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:08.040019 19384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:08.040135 19384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:08.040203 19384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:08.069514 19384 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:35283
I20260928 11:01:08.069566 19511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:35283 every 8 connection(s)
I20260928 11:01:08.070916 19384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:01:08.075544 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:08.073650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:47655)
0928 11:01:08.074357 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:08.074651 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":286,"threads_started":1}
W20260928 11:01:08.076231 19512 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:44131 (0 consecutive failures): Network error: Failed to ping master at 127.3.252.126:44131: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
I20260928 11:01:08.076400 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19384
I20260928 11:01:08.077064 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:08.076703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:46953)
0928 11:01:08.076864 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:08.076950 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 11:01:08.077863 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:08.077532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:38789)
0928 11:01:08.077660 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:08.077732 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 11:01:08.078213 19512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 11:01:08.206374 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:08.205550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:41293)
0928 11:01:08.206039 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:08.206132 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":222,"threads_started":1}
I20260928 11:01:08.642398 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:08.641576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:37835)
0928 11:01:08.642064 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:08.642158 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":224,"threads_started":1}
I20260928 11:01:09.079401 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:09.078669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:48759)
0928 11:01:09.079091 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:09.079180 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":200,"threads_started":1}
I20260928 11:01:09.208031 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:09.207294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:45955)
0928 11:01:09.207778 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:09.207867 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":229,"threads_started":1}
I20260928 11:01:09.644131 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:09.643325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:45477)
0928 11:01:09.643816 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:09.643925 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":216,"threads_started":1}
I20260928 11:01:10.080981 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:10.080247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:51267)
0928 11:01:10.080728 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:10.080815 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
I20260928 11:01:10.209491 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:10.208798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:37997)
0928 11:01:10.209221 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:10.209299 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260928 11:01:10.645895 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:10.645096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:59195)
0928 11:01:10.645578 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:10.645671 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20260928 11:01:11.082571 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:11.081914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:53989)
0928 11:01:11.082328 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:11.082402 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":213,"threads_started":1}
I20260928 11:01:11.211069 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:11.210379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:40355)
0928 11:01:11.210792 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:11.210877 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":194,"threads_started":1}
I20260928 11:01:11.647496 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:11.646770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:55867)
0928 11:01:11.647214 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:11.647301 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260928 11:01:12.084070 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:12.083366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:57113)
0928 11:01:12.083798 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:12.083879 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260928 11:01:12.212725 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:12.211965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:41787)
0928 11:01:12.212454 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:12.212542 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":253,"threads_started":1}
I20260928 11:01:12.649166 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:12.648426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:34463)
0928 11:01:12.648901 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:12.648989 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260928 11:01:13.085680 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:13.084953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:39387)
0928 11:01:13.085421 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:13.085516 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
I20260928 11:01:13.214414 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:13.213635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:38441)
0928 11:01:13.214153 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:13.214250 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":276,"threads_started":1}
I20260928 11:01:13.650758 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:13.650076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:40657)
0928 11:01:13.650499 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:13.650590 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260928 11:01:14.087352 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:14.086637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:36749)
0928 11:01:14.087068 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:14.087165 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260928 11:01:14.216097 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:14.215253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:34391)
0928 11:01:14.215786 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:14.215888 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":252,"threads_started":1}
I20260928 11:01:14.652434 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:14.651637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:35161)
0928 11:01:14.652117 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:14.652218 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":203,"threads_started":1}
I20260928 11:01:15.088986 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:15.088250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:36071)
0928 11:01:15.088698 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:15.088779 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":188,"threads_started":1}
I20260928 11:01:15.217680 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:15.216952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:34849)
0928 11:01:15.217426 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:15.217511 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260928 11:01:15.654060 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:15.653316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:35367)
0928 11:01:15.653777 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:15.653896 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260928 11:01:16.090515 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:16.089817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:39227)
0928 11:01:16.090277 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:16.090359 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":216,"threads_started":1}
I20260928 11:01:16.219293 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:16.218532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:49017)
0928 11:01:16.219027 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:16.219113 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
I20260928 11:01:16.655673 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:16.654899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:54755)
0928 11:01:16.655418 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:16.655503 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":255,"threads_started":1}
I20260928 11:01:17.092075 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:17.091318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:33183)
0928 11:01:17.091812 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:17.091904 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":226,"threads_started":1}
I20260928 11:01:17.221055 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:17.220310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:51569)
0928 11:01:17.220817 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:17.220895 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
I20260928 11:01:17.657246 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:17.656550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:37255)
0928 11:01:17.656995 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:17.657080 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
I20260928 11:01:18.093890 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:18.093099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:37069)
0928 11:01:18.093593 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:18.093685 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":273,"threads_started":1}
I20260928 11:01:18.222671 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:18.221968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:33103)
0928 11:01:18.222378 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:18.222472 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":200,"threads_started":1}
I20260928 11:01:18.658859 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:18.658138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:33649)
0928 11:01:18.658553 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:18.658648 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":203,"threads_started":1}
I20260928 11:01:19.095567 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:19.094850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:33443)
0928 11:01:19.095294 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:19.095396 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":221,"threads_started":1}
I20260928 11:01:19.224351 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:19.223584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:37009)
0928 11:01:19.224092 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:19.224179 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260928 11:01:19.660492 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:19.659770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:48531)
0928 11:01:19.660244 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:19.660333 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260928 11:01:20.097340 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:20.096562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:46799)
0928 11:01:20.097025 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:20.097148 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260928 11:01:20.226028 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:20.225228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:40023)
0928 11:01:20.225708 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:20.225809 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":217,"threads_started":1}
I20260928 11:01:20.662163 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:20.661367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:36841)
0928 11:01:20.661882 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:20.661981 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":265,"threads_started":1}
I20260928 11:01:21.099129 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:21.098222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:49841)
0928 11:01:21.098750 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:21.098922 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":238,"threads_started":1}
I20260928 11:01:21.227602 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:21.226875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:56875)
0928 11:01:21.227331 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:21.227419 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":202,"threads_started":1}
I20260928 11:01:21.663720 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:21.663006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:35697)
0928 11:01:21.663424 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:21.663509 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
I20260928 11:01:22.100618 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:22.099936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:59969)
0928 11:01:22.100366 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:22.100447 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":219,"threads_started":1}
I20260928 11:01:22.229588 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:22.228746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.65:55975)
0928 11:01:22.229241 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:22.229357 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
I20260928 11:01:22.665477 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:22.664711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.66:35157)
0928 11:01:22.665182 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:22.665301 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":263,"threads_started":1}
I20260928 11:01:23.084048 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 19087
I20260928 11:01:23.102334 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:23.101583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:44131 (local address 127.3.252.67:40687)
0928 11:01:23.102040 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:23.102141 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
I20260928 11:01:23.105216 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:23.085116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.3.252.65:42873)
0928 11:01:23.085706 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:23.085725 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:23.085769 (+ 44us) server_negotiation.cc:407] Connection header received
0928 11:01:23.087797 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:23.087807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:23.088215 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:23.088514 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:23.089868 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:23.091108 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:23.092372 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:23.093047 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:23.093206 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:23.099338 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:23.099347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:23.099364 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:23.099404 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:23.102174 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:23.102894 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:23.102902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:23.102908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:23.103097 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:23.103398 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:23.103401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:23.103403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:23.103947 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:23.104192 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:23.104537 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 11:01:23.104927 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260928 11:01:23.106593 19245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:23.189777 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 19087
I20260928 11:01:23.204380 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 19253
I20260928 11:01:23.221207 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:23.205338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.3.252.66:35199)
0928 11:01:23.205888 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:23.205902 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:23.205938 (+ 36us) server_negotiation.cc:407] Connection header received
0928 11:01:23.207458 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:23.207466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:23.207870 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:23.208138 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:23.209366 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:23.210656 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:23.211901 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:23.212550 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:23.212674 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:23.215224 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:23.215232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:23.215247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:23.215302 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:23.218277 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:23.218954 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:23.218960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:23.218966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:23.219120 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:23.219454 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:23.219458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:23.219460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:23.219978 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:23.220209 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:23.220583 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 11:01:23.220917 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260928 11:01:23.222698 19376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:23.307089 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 19253
I20260928 11:01:23.321730 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 19384
I20260928 11:01:23.338572 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:23.322684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.3.252.67:35283)
0928 11:01:23.323250 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:23.323265 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:23.323305 (+ 40us) server_negotiation.cc:407] Connection header received
0928 11:01:23.324882 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:23.324892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:23.325304 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:23.325604 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:23.327029 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:23.328260 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:23.329513 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:23.330196 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:23.330340 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:23.332835 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:23.332844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:23.332859 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:23.332898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:23.335600 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:23.336211 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:23.336217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:23.336224 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:23.336390 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:23.336673 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:23.336676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:23.336678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:23.337305 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:23.337548 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:23.337878 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 11:01:23.338281 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":250,"threads_started":1}
I20260928 11:01:23.339870 19507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:23.421875 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 19384
2026-09-28T11:01:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24147 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:01:23 dist-test-slave-djjw krb5kdc[19569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:01:23 dist-test-slave-djjw krb5kdc[19569](info): set up 2 sockets
Sep 28 11:01:23 dist-test-slave-djjw krb5kdc[19569](info): commencing operation
krb5kdc: starting...
W20260928 11:01:24.430415 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.010s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:01:24 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:01:24.655349 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:37019
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:24.970754 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:24.971055 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:24.971153 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:24.978497 19582 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 11:01:24.978600 19582 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 11:01:24.978636 19582 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 11:01:24.978705 19582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:01:24.978758 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:24.978823 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:24.978866 19582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:01:24.978931 19582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:01:24.988803 19582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:24.990046 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:24.991278 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:25.002856 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:25.002766 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:25.002851 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:25.003479 19582 server_base.cc:1034] running on GCE node
I20260928 11:01:25.004235 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:25.005683 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:25.006916 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790593285006850 us; error 43 us; skew 500 ppm
I20260928 11:01:25.007253 19582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:25 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593285, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:25.010303 19582 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:01:25.011759 19582 webserver.cc:460] Webserver started at http://127.3.252.126:38575/ using document root <none> and password file <none>
I20260928 11:01:25.012491 19582 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:25.012598 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:25.012915 19582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:25.015980 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "7b8001367dfc41ce947c16ab4a393483"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "411cc0a44e58eb72fa68f89bb0ce5ac0"
server_key_iv: "4e83e17e6dafc49d37e197fdf521c075"
server_key_version: "encryptionkey@0"
I20260928 11:01:25.016767 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "7b8001367dfc41ce947c16ab4a393483"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "411cc0a44e58eb72fa68f89bb0ce5ac0"
server_key_iv: "4e83e17e6dafc49d37e197fdf521c075"
server_key_version: "encryptionkey@0"
I20260928 11:01:25.022135 19582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 11:01:25.025753 19597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:25.027211 19582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:01:25.027391 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "7b8001367dfc41ce947c16ab4a393483"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "411cc0a44e58eb72fa68f89bb0ce5ac0"
server_key_iv: "4e83e17e6dafc49d37e197fdf521c075"
server_key_version: "encryptionkey@0"
I20260928 11:01:25.027606 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:25.069376 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:25.073207 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:25.073455 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:25.090145 19582 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:37019
I20260928 11:01:25.090164 19648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:37019 every 8 connection(s)
I20260928 11:01:25.091415 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:01:25.094998 19649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:25.097028 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19582
I20260928 11:01:25.097218 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:01:25.097863 4081 external_mini_cluster.cc:1428] Setting key 6b36ea8e6472c158d042d2b19ae470ea
I20260928 11:01:25.106462 19649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Bootstrap starting.
I20260928 11:01:25.110204 19649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:25.111493 19649 log.cc:824] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Log is configured to *not* fsync() on all Append() calls
Sep 28 11:01:25 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:25.114632 19649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: No bootstrap required, opened a new log
I20260928 11:01:25.121047 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:25.099805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.3.252.126:37019)
0928 11:01:25.100540 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:25.100552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:25.100593 (+ 41us) server_negotiation.cc:407] Connection header received
0928 11:01:25.102363 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:25.102389 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:25.102808 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:25.103139 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:25.104928 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:25.106361 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:25.108195 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:25.108894 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:25.109022 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:25.113897 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:25.113907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:25.113923 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:25.113965 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:25.116957 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:25.117753 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:25.117759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:25.117765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:25.117993 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:25.118375 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:25.118378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:25.118380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:25.118910 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:25.119185 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:25.119595 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 11:01:25.119970 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":339,"threads_started":1}
I20260928 11:01:25.123137 19649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:25.123476 19649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:25.123581 19649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8001367dfc41ce947c16ab4a393483, State: Initialized, Role: FOLLOWER
I20260928 11:01:25.124109 19649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:25.124318 19649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:01:25.124413 19649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:01:25.124529 19649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:25.126767 19649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:25.127259 19649 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8001367dfc41ce947c16ab4a393483; no voters:
I20260928 11:01:25.127780 19649 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:01:25.127904 19654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:01:25.128561 19654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8001367dfc41ce947c16ab4a393483, State: Running, Role: LEADER
I20260928 11:01:25.129097 19654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:25.129703 19649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:01:25.131771 19655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b8001367dfc41ce947c16ab4a393483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } } }
I20260928 11:01:25.132140 19655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:25.133723 19660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:01:25.133796 19656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8001367dfc41ce947c16ab4a393483. Latest consensus state: current_term: 1 leader_uuid: "7b8001367dfc41ce947c16ab4a393483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } } }
I20260928 11:01:25.134219 19656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:25.141533 19660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:01:25.149606 19660 catalog_manager.cc:1347] Generated new cluster ID: 6cd9a27a3ed040f2b8e4206e76867738
I20260928 11:01:25.149718 19660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:01:25.159476 19660 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:01:25.161425 19660 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:01:25.177122 19660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Generated new TSK 0
I20260928 11:01:25.178256 19660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 11:01:25.229017 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:37019
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:25.546356 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:25.546636 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:25.546761 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:25.554083 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:25.554194 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:25.554340 19675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:01:25.564080 19675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:25.565351 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:25.566627 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:25.578944 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:25.579010 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:25.578979 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:25.579839 19675 server_base.cc:1034] running on GCE node
I20260928 11:01:25.580420 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:25.581296 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:25.582492 19675 hybrid_clock.cc:648] HybridClock initialized: now 1790593285582436 us; error 46 us; skew 500 ppm
I20260928 11:01:25.582778 19675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:25 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593285, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:25.585798 19675 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:01:25.587275 19675 webserver.cc:460] Webserver started at http://127.3.252.65:46509/ using document root <none> and password file <none>
I20260928 11:01:25.587922 19675 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:25.588028 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:25.588315 19675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:25.591238 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "5cdd07284bdb46d198bffda9961855cb"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "a67b097e9e10e29f54882e2a007ae381"
server_key_iv: "7ac4844cfc39d47f65409fe3222ed9ca"
server_key_version: "encryptionkey@0"
I20260928 11:01:25.591938 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "5cdd07284bdb46d198bffda9961855cb"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "a67b097e9e10e29f54882e2a007ae381"
server_key_iv: "7ac4844cfc39d47f65409fe3222ed9ca"
server_key_version: "encryptionkey@0"
I20260928 11:01:25.596765 19675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 11:01:25.600136 19690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:25.601567 19675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:01:25.601768 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "5cdd07284bdb46d198bffda9961855cb"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "a67b097e9e10e29f54882e2a007ae381"
server_key_iv: "7ac4844cfc39d47f65409fe3222ed9ca"
server_key_version: "encryptionkey@0"
I20260928 11:01:25.601998 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:25.617116 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:25.620891 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:25.621133 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:25.622080 19675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:25.623302 19675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:25.623373 19675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:25.623492 19675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:25.623544 19675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:25.652585 19675 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:33585
I20260928 11:01:25.652645 19802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:33585 every 8 connection(s)
I20260928 11:01:25.653884 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:01:25.660481 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19675
I20260928 11:01:25.660710 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:01:25.661490 4081 external_mini_cluster.cc:1428] Setting key 8c512354b43ac8b57ea204002a50c9ab
Sep 28 11:01:25 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593285, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:25.674424 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:25.656271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:44173 (local address 127.3.252.126:37019)
0928 11:01:25.656698 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:25.656702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:25.657596 (+ 894us) server_negotiation.cc:407] Connection header received
0928 11:01:25.659427 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:25.659433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:25.659522 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:25.659615 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:25.661578 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:25.662466 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:25.664476 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:25.664866 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:25.664959 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:25.668967 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:25.668972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:25.668975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:25.669007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:25.671536 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:25.672210 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:25.672215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:25.672218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:25.672362 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:25.672719 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:25.672723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:25.672726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:25.673050 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:25.673180 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:25.673823 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 11:01:25.674054 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260928 11:01:25.675257 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:25.656703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:44173)
0928 11:01:25.657435 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:25.657473 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:25.659155 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:25.659857 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:25.659866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:25.660581 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:25.661387 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:25.661412 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:25.662613 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:25.662617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:25.664324 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:25.664350 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:25.665075 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:25.665082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:25.665369 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:25.666110 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:25.666139 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:25.668734 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:25.671665 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:25.671672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:25.671686 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:25.672102 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:25.672450 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:25.672453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:25.672455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:25.672633 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:25.673154 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:25.673160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:25.673498 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 11:01:25.673726 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 11:01:25.674155 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":256,"threads_started":1}
I20260928 11:01:25.676822 19803 heartbeater.cc:347] Connected to a master server at 127.3.252.126:37019
I20260928 11:01:25.677145 19803 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:25.677979 19803 heartbeater.cc:511] Master 127.3.252.126:37019 requested a full tablet report, sending...
I20260928 11:01:25.680133 19614 ts_manager.cc:194] Registered new tserver with Master: 5cdd07284bdb46d198bffda9961855cb (127.3.252.65:33585)
I20260928 11:01:25.681622 19614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:44173
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 11:01:25.712631 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:37019
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:26.022396 19809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:26.022683 19809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:26.022817 19809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:26.030452 19809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:26.030562 19809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:26.030707 19809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:01:26.040679 19809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:26.041985 19809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:26.043224 19809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:26.055495 19814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:26.055444 19815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:26.055495 19817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:26.056329 19809 server_base.cc:1034] running on GCE node
I20260928 11:01:26.056936 19809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:26.057859 19809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:26.059049 19809 hybrid_clock.cc:648] HybridClock initialized: now 1790593286058996 us; error 48 us; skew 500 ppm
I20260928 11:01:26.059330 19809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:26 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593286, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:26.062219 19809 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:01:26.063685 19809 webserver.cc:460] Webserver started at http://127.3.252.66:32985/ using document root <none> and password file <none>
I20260928 11:01:26.064337 19809 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:26.064450 19809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:26.064745 19809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:26.067655 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "e3c014f68f534cfb95da27367857b48b"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "493304c15623331923f20d02d43d2ad5"
server_key_iv: "2c926a45fae26c058dea5d31428f88c0"
server_key_version: "encryptionkey@0"
I20260928 11:01:26.068403 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "e3c014f68f534cfb95da27367857b48b"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "493304c15623331923f20d02d43d2ad5"
server_key_iv: "2c926a45fae26c058dea5d31428f88c0"
server_key_version: "encryptionkey@0"
I20260928 11:01:26.073403 19809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 11:01:26.076731 19824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:26.078105 19809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:01:26.078245 19809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "e3c014f68f534cfb95da27367857b48b"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "493304c15623331923f20d02d43d2ad5"
server_key_iv: "2c926a45fae26c058dea5d31428f88c0"
server_key_version: "encryptionkey@0"
I20260928 11:01:26.078423 19809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:26.101116 19809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:26.106403 19809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:26.106695 19809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:26.107761 19809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:26.109325 19809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:26.109409 19809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:26.109515 19809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:26.109572 19809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:26.142640 19809 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:43971
I20260928 11:01:26.142647 19936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:43971 every 8 connection(s)
I20260928 11:01:26.143953 19809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:01:26.153504 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19809
I20260928 11:01:26.153709 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:01:26.154421 4081 external_mini_cluster.cc:1428] Setting key 63192eeb7c09193309d82728fe1700ff
Sep 28 11:01:26 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593286, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:26.164788 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.146538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:50599 (local address 127.3.252.126:37019)
0928 11:01:26.146710 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.146715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.147838 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 11:01:26.149569 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.149573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.149642 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.149736 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:26.151232 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.151950 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.153522 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.154276 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.154415 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.158692 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:26.158698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:26.158703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:26.158741 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:26.161765 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.162544 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.162550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.162553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.162702 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.163095 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.163099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.163102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.163436 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:26.163582 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.164251 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.164481 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 11:01:26.165953 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.146999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:50599)
0928 11:01:26.147680 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:26.147718 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:26.149335 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:26.149981 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:26.149989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:26.150503 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:26.151065 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.151082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.152062 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.152065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.153371 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.153384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.154521 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.154530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.154780 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.155866 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:26.155902 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:26.158461 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:26.161945 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:26.161952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:26.161968 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:26.162438 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:26.162806 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:26.162810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:26.162812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:26.162998 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:26.163560 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:26.163567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:26.163911 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 11:01:26.164140 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 11:01:26.164539 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":261,"threads_started":1}
I20260928 11:01:26.167351 19937 heartbeater.cc:347] Connected to a master server at 127.3.252.126:37019
I20260928 11:01:26.167655 19937 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:26.168395 19937 heartbeater.cc:511] Master 127.3.252.126:37019 requested a full tablet report, sending...
I20260928 11:01:26.169904 19614 ts_manager.cc:194] Registered new tserver with Master: e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
I20260928 11:01:26.170729 19614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:50599
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 11:01:26.206099 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:37019
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:26.523102 19942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:26.523401 19942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:26.523524 19942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:26.530766 19942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:26.530885 19942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:26.531041 19942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:01:26.540907 19942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:26.542233 19942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:26.543527 19942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:26.557346 19947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:26.557272 19950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:26.557304 19948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:26.558357 19942 server_base.cc:1034] running on GCE node
I20260928 11:01:26.558965 19942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:26.559862 19942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:26.561070 19942 hybrid_clock.cc:648] HybridClock initialized: now 1790593286561016 us; error 43 us; skew 500 ppm
I20260928 11:01:26.561362 19942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:26 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593286, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:26.564424 19942 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:01:26.565927 19942 webserver.cc:460] Webserver started at http://127.3.252.67:35943/ using document root <none> and password file <none>
I20260928 11:01:26.566605 19942 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:26.566839 19942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:26.567188 19942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:01:26.571681 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "cefbcea617ea4016a72abd5a8df9e859"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "744e19e566c7f2501a195ab97bec596d"
server_key_iv: "4bb3db4b638d6a7dd26c078864e69fa8"
server_key_version: "encryptionkey@0"
I20260928 11:01:26.572925 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "cefbcea617ea4016a72abd5a8df9e859"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "744e19e566c7f2501a195ab97bec596d"
server_key_iv: "4bb3db4b638d6a7dd26c078864e69fa8"
server_key_version: "encryptionkey@0"
I20260928 11:01:26.580438 19942 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 11:01:26.585179 19957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:26.586969 19942 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 11:01:26.587157 19942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "cefbcea617ea4016a72abd5a8df9e859"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "744e19e566c7f2501a195ab97bec596d"
server_key_iv: "4bb3db4b638d6a7dd26c078864e69fa8"
server_key_version: "encryptionkey@0"
I20260928 11:01:26.587404 19942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:26.603186 19942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:26.608398 19942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:26.608685 19942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:26.609658 19942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:26.611055 19942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:26.611135 19942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:26.611253 19942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:26.611304 19942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:26.639340 19942 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:37811
I20260928 11:01:26.639384 20069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:37811 every 8 connection(s)
I20260928 11:01:26.640538 19942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:01:26.646786 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 19942
I20260928 11:01:26.646943 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:01:26.647357 4081 external_mini_cluster.cc:1428] Setting key 5e6433cf4cedd87a3033709351c67347
Sep 28 11:01:26 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593286, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:26.659513 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.642922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:38121 (local address 127.3.252.126:37019)
0928 11:01:26.643085 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.643089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.644222 (+ 1133us) server_negotiation.cc:407] Connection header received
0928 11:01:26.646077 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.646083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.646163 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.646248 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:26.647733 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.648678 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.650595 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.650992 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.651095 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.654501 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:26.654505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:26.654509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:26.654541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:26.657019 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.657670 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.657674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.657676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.657789 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.658157 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.658160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.658162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.658433 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:26.658538 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.659093 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.659261 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 11:01:26.660434 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.643413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:38121)
0928 11:01:26.644074 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:26.644109 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:26.645792 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:26.646475 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:26.646483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:26.647022 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:26.647581 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.647600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.648810 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.648821 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.650465 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.650480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.651188 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.651195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.651415 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.652001 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:26.652023 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:26.654289 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:26.657139 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:26.657146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:26.657156 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:26.657575 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:26.657896 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:26.657899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:26.657901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:26.658072 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:26.658524 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:26.658530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:26.658813 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 11:01:26.659023 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 11:01:26.659396 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":251,"threads_started":1}
I20260928 11:01:26.661795 20070 heartbeater.cc:347] Connected to a master server at 127.3.252.126:37019
I20260928 11:01:26.662107 20070 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:26.662789 20070 heartbeater.cc:511] Master 127.3.252.126:37019 requested a full tablet report, sending...
I20260928 11:01:26.664216 19613 ts_manager.cc:194] Registered new tserver with Master: cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811)
I20260928 11:01:26.665059 19613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:38121
I20260928 11:01:26.674608 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:01:26.685103 19803 heartbeater.cc:503] Master 127.3.252.126:37019 was elected leader, sending a full tablet report...
I20260928 11:01:26.691151 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.678830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.3.252.126:37019)
0928 11:01:26.678985 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.678991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.679391 (+ 400us) server_negotiation.cc:407] Connection header received
0928 11:01:26.679550 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.679555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.679622 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.679699 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:26.681046 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.681870 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.683165 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.683558 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.683658 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.686182 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:26.686187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:26.686190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:26.686226 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:26.688638 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.689260 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.689266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.689270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.689400 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.689735 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.689741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.689744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.690099 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:26.690316 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.690632 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.690844 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 11:01:26.694722 19613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:01:26.701387 19613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:01:26.724704 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.713120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:37811 (local address 127.0.0.1:51688)
0928 11:01:26.713971 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:26.714018 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:26.714179 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:26.718071 (+ 3892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:26.718079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:26.718107 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:26.718464 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.718476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.720291 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.720298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.722251 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.722263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.723988 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.723992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.724129 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.724163 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 11:01:26.724260 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 11:01:26.724406 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":48,"thread_start_us":458,"threads_started":1}
I20260928 11:01:26.724822 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.713340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.3.252.67:37811)
0928 11:01:26.717509 (+ 4169us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.717517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.717548 (+ 31us) server_negotiation.cc:407] Connection header received
0928 11:01:26.717634 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.717641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.717878 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.718640 (+ 762us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:26.718905 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.720148 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.722424 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.723821 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.723952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.724192 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.724304 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.724478 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4004,"thread_start_us":3838,"threads_started":1}
I20260928 11:01:26.726039 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.714443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:43971 (local address 127.0.0.1:44976)
0928 11:01:26.716716 (+ 2273us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:26.716732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:26.716833 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:26.717471 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:26.717476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:26.717500 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:26.718143 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.718155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.719466 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.719471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.722226 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.722236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.725199 (+ 2963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.725205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.725376 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.725409 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 11:01:26.725492 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 11:01:26.725631 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2158,"mutex_wait_us":157,"thread_start_us":2023,"threads_started":1}
I20260928 11:01:26.726613 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.714897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.3.252.66:43971)
0928 11:01:26.716314 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.716321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.716944 (+ 623us) server_negotiation.cc:407] Connection header received
0928 11:01:26.717041 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.717047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.717285 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.717466 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:26.718368 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.719343 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.722365 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.723821 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.723952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.724167 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.726238 (+ 2071us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.726388 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":1103,"threads_started":1}
I20260928 11:01:26.727017 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.713401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:43446)
0928 11:01:26.715977 (+ 2576us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:26.715995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:26.716116 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:26.716868 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:26.716876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:26.716912 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:26.717820 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.717862 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.719728 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.719733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.722198 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.722220 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.726263 (+ 4043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.726269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.726429 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.726460 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 11:01:26.726530 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 11:01:26.726653 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2448,"mutex_wait_us":122,"thread_start_us":2313,"threads_started":1}
I20260928 11:01:26.727931 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.714044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.3.252.65:33585)
0928 11:01:26.715568 (+ 1524us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.715578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.716237 (+ 659us) server_negotiation.cc:407] Connection header received
0928 11:01:26.716340 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.716348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.716615 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.716832 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:26.718074 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.719605 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.724259 (+ 4654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.725924 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.726057 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.727386 (+ 1329us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.727472 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.727627 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1229,"threads_started":1}
I20260928 11:01:26.728919 20005 tablet_service.cc:1467] Processing CreateTablet for tablet f6f69f2eb37e45a1b6a141ccd07ec252 (DEFAULT_TABLE table=test-table [id=5a34a549261048b1a7fbaa9de013850d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:26.729724 19872 tablet_service.cc:1467] Processing CreateTablet for tablet f6f69f2eb37e45a1b6a141ccd07ec252 (DEFAULT_TABLE table=test-table [id=5a34a549261048b1a7fbaa9de013850d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:26.730486 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f69f2eb37e45a1b6a141ccd07ec252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:26.731113 19872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f69f2eb37e45a1b6a141ccd07ec252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:26.731536 19738 tablet_service.cc:1467] Processing CreateTablet for tablet f6f69f2eb37e45a1b6a141ccd07ec252 (DEFAULT_TABLE table=test-table [id=5a34a549261048b1a7fbaa9de013850d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:26.732836 19738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f69f2eb37e45a1b6a141ccd07ec252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:26.741744 20088 tablet_bootstrap.cc:492] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Bootstrap starting.
I20260928 11:01:26.744519 20089 tablet_bootstrap.cc:492] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Bootstrap starting.
I20260928 11:01:26.746205 20090 tablet_bootstrap.cc:492] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Bootstrap starting.
I20260928 11:01:26.746744 20088 tablet_bootstrap.cc:654] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:26.747768 20089 tablet_bootstrap.cc:654] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:26.747992 20088 log.cc:824] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:26.748879 20089 log.cc:824] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:26.749007 20090 tablet_bootstrap.cc:654] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:26.750072 20090 log.cc:824] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:26.751691 20088 tablet_bootstrap.cc:492] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: No bootstrap required, opened a new log
I20260928 11:01:26.751943 20088 ts_tablet_manager.cc:1397] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 11:01:26.753158 20089 tablet_bootstrap.cc:492] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: No bootstrap required, opened a new log
I20260928 11:01:26.753418 20089 ts_tablet_manager.cc:1397] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260928 11:01:26.755498 20090 tablet_bootstrap.cc:492] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: No bootstrap required, opened a new log
I20260928 11:01:26.755769 20090 ts_tablet_manager.cc:1397] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260928 11:01:26.760412 20089 raft_consensus.cc:348] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.760772 20089 raft_consensus.cc:374] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:26.760880 20089 raft_consensus.cc:729] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cefbcea617ea4016a72abd5a8df9e859, State: Initialized, Role: FOLLOWER
I20260928 11:01:26.761451 20089 consensus_queue.cc:260] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.761930 20088 raft_consensus.cc:348] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.762429 20088 raft_consensus.cc:374] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:26.762507 20070 heartbeater.cc:503] Master 127.3.252.126:37019 was elected leader, sending a full tablet report...
I20260928 11:01:26.762573 20088 raft_consensus.cc:729] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c014f68f534cfb95da27367857b48b, State: Initialized, Role: FOLLOWER
I20260928 11:01:26.763196 20089 ts_tablet_manager.cc:1428] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260928 11:01:26.763203 20088 consensus_queue.cc:260] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.765339 20088 ts_tablet_manager.cc:1428] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 11:01:26.765452 19937 heartbeater.cc:503] Master 127.3.252.126:37019 was elected leader, sending a full tablet report...
I20260928 11:01:26.765882 20090 raft_consensus.cc:348] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.766234 20090 raft_consensus.cc:374] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:26.766327 20090 raft_consensus.cc:729] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdd07284bdb46d198bffda9961855cb, State: Initialized, Role: FOLLOWER
I20260928 11:01:26.766990 20090 consensus_queue.cc:260] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.769654 20096 raft_consensus.cc:482] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:01:26.769935 20096 raft_consensus.cc:504] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.770474 20090 ts_tablet_manager.cc:1428] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 11:01:26.771629 20096 leader_election.cc:288] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
I20260928 11:01:26.778218 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.771949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:43911 (local address 127.3.252.67:37811)
0928 11:01:26.772152 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.772159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.772615 (+ 456us) server_negotiation.cc:407] Connection header received
0928 11:01:26.772902 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.772908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.772989 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.773046 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:26.773661 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.774432 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.776444 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.777360 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.777503 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.777575 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.777867 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.777988 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 11:01:26.778354 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.771928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:37811 (local address 127.3.252.65:43911)
0928 11:01:26.772533 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:26.772550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:26.772677 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:26.773228 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:26.773233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:26.773255 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:26.773523 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.773530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.774554 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.774560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.776281 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.776294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.777542 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.777548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.777686 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.777709 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 11:01:26.777800 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 11:01:26.777962 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":50,"thread_start_us":364,"threads_started":1}
I20260928 11:01:26.778579 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.772807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:49935 (local address 127.3.252.66:43971)
0928 11:01:26.772992 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.772998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.773276 (+ 278us) server_negotiation.cc:407] Connection header received
0928 11:01:26.773444 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.773449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.773509 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.773577 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:26.774099 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.774765 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.776522 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.777514 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.777646 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.777709 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.778212 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.778345 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 11:01:26.778645 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.772706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:43971 (local address 127.3.252.65:49935)
0928 11:01:26.773190 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:26.773204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:26.773326 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:26.773602 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:26.773607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:26.773628 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:26.773970 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.773980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.774884 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.774890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.776395 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:26.776410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.777896 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.777902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:26.778055 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.778092 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 11:01:26.778173 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 11:01:26.778294 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260928 11:01:26.779572 19892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f69f2eb37e45a1b6a141ccd07ec252" candidate_uuid: "5cdd07284bdb46d198bffda9961855cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c014f68f534cfb95da27367857b48b" is_pre_election: true
I20260928 11:01:26.779572 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f69f2eb37e45a1b6a141ccd07ec252" candidate_uuid: "5cdd07284bdb46d198bffda9961855cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cefbcea617ea4016a72abd5a8df9e859" is_pre_election: true
I20260928 11:01:26.780138 20025 raft_consensus.cc:2454] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cdd07284bdb46d198bffda9961855cb in term 0.
I20260928 11:01:26.780138 19892 raft_consensus.cc:2454] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cdd07284bdb46d198bffda9961855cb in term 0.
I20260928 11:01:26.780807 19692 leader_election.cc:302] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb, cefbcea617ea4016a72abd5a8df9e859; no voters:
I20260928 11:01:26.781232 20096 raft_consensus.cc:2781] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:01:26.781437 20096 raft_consensus.cc:482] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:01:26.781553 20096 raft_consensus.cc:3037] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:26.783768 20096 raft_consensus.cc:504] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.784332 20096 leader_election.cc:288] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 1 election: Requested vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
I20260928 11:01:26.784818 19892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f69f2eb37e45a1b6a141ccd07ec252" candidate_uuid: "5cdd07284bdb46d198bffda9961855cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c014f68f534cfb95da27367857b48b"
I20260928 11:01:26.784818 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f69f2eb37e45a1b6a141ccd07ec252" candidate_uuid: "5cdd07284bdb46d198bffda9961855cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cefbcea617ea4016a72abd5a8df9e859"
I20260928 11:01:26.785028 19892 raft_consensus.cc:3037] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:26.785028 20025 raft_consensus.cc:3037] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:26.788131 19892 raft_consensus.cc:2454] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cdd07284bdb46d198bffda9961855cb in term 1.
I20260928 11:01:26.788131 20025 raft_consensus.cc:2454] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cdd07284bdb46d198bffda9961855cb in term 1.
I20260928 11:01:26.788498 19692 leader_election.cc:302] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb, e3c014f68f534cfb95da27367857b48b; no voters:
I20260928 11:01:26.788761 20096 raft_consensus.cc:2781] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:01:26.788921 20096 raft_consensus.cc:686] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 1 LEADER]: Becoming Leader. State: Replica: 5cdd07284bdb46d198bffda9961855cb, State: Running, Role: LEADER
I20260928 11:01:26.789388 20096 consensus_queue.cc:237] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } }
I20260928 11:01:26.793812 19613 catalog_manager.cc:5495] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cdd07284bdb46d198bffda9961855cb (127.3.252.65). New cstate: current_term: 1 leader_uuid: "5cdd07284bdb46d198bffda9961855cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } health_report { overall_health: UNKNOWN } } }
I20260928 11:01:26.824123 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.819443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.3.252.65:33585)
0928 11:01:26.819610 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.819615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.819633 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:01:26.819803 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.819807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.819908 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.820030 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:01:26.820308 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.821011 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.822325 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.822721 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.822808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.823200 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:01:26.823701 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:01:26.823807 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.823860 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.823926 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 11:01:26.831949 19892 raft_consensus.cc:1261] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 1 FOLLOWER]: Refusing update from remote peer 5cdd07284bdb46d198bffda9961855cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:26.831996 20025 raft_consensus.cc:1261] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Refusing update from remote peer 5cdd07284bdb46d198bffda9961855cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:26.832870 20097 consensus_queue.cc:1035] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:26.833197 20096 consensus_queue.cc:1035] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:26.842013 20102 mvcc.cc:205] Tried to move back new op lower bound from 7334270102854090752 to 7334270102693048320. Current Snapshot: MvccSnapshot[applied={T|T < 7334270102854090752}]
I20260928 11:01:26.843267 20103 mvcc.cc:205] Tried to move back new op lower bound from 7334270102854090752 to 7334270102693048320. Current Snapshot: MvccSnapshot[applied={T|T < 7334270102854090752}]
I20260928 11:01:26.859467 19614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:26.859766 19614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:26.863548 19614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Sending DeleteTablet for 3 replicas of tablet f6f69f2eb37e45a1b6a141ccd07ec252
I20260928 11:01:26.864754 19738 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f69f2eb37e45a1b6a141ccd07ec252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:26 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:43446
I20260928 11:01:26.864842 20005 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f69f2eb37e45a1b6a141ccd07ec252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:26 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:51688
I20260928 11:01:26.864984 19872 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f69f2eb37e45a1b6a141ccd07ec252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:26 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:44976
I20260928 11:01:26.866030 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 19675
I20260928 11:01:26.875741 20113 tablet_replica.cc:330] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: stopping tablet replica
I20260928 11:01:26.877002 20113 raft_consensus.cc:2229] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:26.877545 20112 tablet_replica.cc:330] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: stopping tablet replica
I20260928 11:01:26.877727 20113 raft_consensus.cc:2258] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:26.878314 20114 tablet_replica.cc:330] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: stopping tablet replica
I20260928 11:01:26.878479 20112 raft_consensus.cc:2229] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:01:26.879179 20112 raft_consensus.cc:2258] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:26.879242 20114 raft_consensus.cc:2229] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:26.880741 20113 ts_tablet_manager.cc:1905] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:26.881124 20112 ts_tablet_manager.cc:1905] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:26.881255 20114 raft_consensus.cc:2258] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:26.883368 20114 ts_tablet_manager.cc:1905] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:26.888006 20113 ts_tablet_manager.cc:1918] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:26.888221 20113 log.cc:1196] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/f6f69f2eb37e45a1b6a141ccd07ec252
I20260928 11:01:26.888337 20112 ts_tablet_manager.cc:1918] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:26.888769 20113 ts_tablet_manager.cc:1939] T f6f69f2eb37e45a1b6a141ccd07ec252 P cefbcea617ea4016a72abd5a8df9e859: Deleting consensus metadata
I20260928 11:01:26.890367 20114 ts_tablet_manager.cc:1918] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:26.890784 20114 log.cc:1196] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/f6f69f2eb37e45a1b6a141ccd07ec252
I20260928 11:01:26.890707 19599 catalog_manager.cc:4841] TS cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811): tablet f6f69f2eb37e45a1b6a141ccd07ec252 (table test-table [id=5a34a549261048b1a7fbaa9de013850d]) successfully deleted
I20260928 11:01:26.891391 20114 ts_tablet_manager.cc:1939] T f6f69f2eb37e45a1b6a141ccd07ec252 P e3c014f68f534cfb95da27367857b48b: Deleting consensus metadata
I20260928 11:01:26.891728 20112 log.cc:1196] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/f6f69f2eb37e45a1b6a141ccd07ec252
I20260928 11:01:26.892403 20112 ts_tablet_manager.cc:1939] T f6f69f2eb37e45a1b6a141ccd07ec252 P 5cdd07284bdb46d198bffda9961855cb: Deleting consensus metadata
I20260928 11:01:26.893245 19599 catalog_manager.cc:4841] TS e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971): tablet f6f69f2eb37e45a1b6a141ccd07ec252 (table test-table [id=5a34a549261048b1a7fbaa9de013850d]) successfully deleted
I20260928 11:01:26.894161 19598 catalog_manager.cc:4841] TS 5cdd07284bdb46d198bffda9961855cb (127.3.252.65:33585): tablet f6f69f2eb37e45a1b6a141ccd07ec252 (table test-table [id=5a34a549261048b1a7fbaa9de013850d]) successfully deleted
Sep 28 11:01:26 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 11:01:26.905668 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:26.867245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.3.252.65:33585)
0928 11:01:26.867438 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:26.867445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:26.889064 (+ 21619us) server_negotiation.cc:407] Connection header received
0928 11:01:26.889386 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:26.889391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:26.889477 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:26.889549 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:26.891622 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.892854 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.894732 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:26.895169 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:26.895273 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:26.899882 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:26.899893 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:26.899907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:26.899944 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:26.902809 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.903622 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.903628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.903630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.903770 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:26.904178 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:26.904181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:26.904183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:26.904512 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:26.904721 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:26.905148 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 11:01:26.905353 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 11:01:26.906426 19798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:27.008378 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 19675
I20260928 11:01:27.023437 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 19809
Sep 28 11:01:27 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 11:01:27.038121 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:27.024326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.3.252.66:43971)
0928 11:01:27.024501 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:27.024505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:27.024524 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:01:27.024612 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:27.024615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:27.024685 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:27.024765 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:27.026022 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:27.026765 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:27.028104 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:27.028508 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:27.028607 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:27.032771 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:27.032786 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:27.032802 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:27.032836 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:27.035529 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:27.036236 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:27.036243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:27.036246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:27.036376 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:27.036682 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:27.036686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:27.036688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:27.037080 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:27.037307 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:27.037594 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 11:01:27.037762 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 11:01:27.038789 19932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:27.126055 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 19809
I20260928 11:01:27.139967 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 19942
Sep 28 11:01:27 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 11:01:27.153985 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:27.140648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.3.252.67:37811)
0928 11:01:27.140841 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:27.140845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:27.140883 (+ 38us) server_negotiation.cc:407] Connection header received
0928 11:01:27.140938 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:27.140941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:27.141013 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:27.141085 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:27.142384 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:27.143120 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:27.144357 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:27.144797 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:27.144892 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:27.148870 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:27.148888 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:27.148904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:27.148939 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:27.151564 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:27.152264 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:27.152271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:27.152274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:27.152407 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:27.152718 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:27.152721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:27.152723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:27.153066 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:27.153270 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:27.153494 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 11:01:27.153656 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 11:01:27.154623 20065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:27.242385 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 19942
I20260928 11:01:27.255908 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 19582
I20260928 11:01:27.257058 19644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:27.367722 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 19582
I20260928 11:01:47.385689 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:37019
--webserver_interface=127.3.252.126
--webserver_port=38575
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:01:47.704449 20120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:47.704731 20120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:47.704859 20120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:47.711987 20120 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 11:01:47.712082 20120 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 11:01:47.712126 20120 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 11:01:47.712193 20120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:01:47.712265 20120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:47.712354 20120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:47.712409 20120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:01:47.712481 20120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:01:47.722252 20120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=38575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:47.723570 20120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:47.724857 20120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:47.746097 20127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:47.746132 20129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:47.746044 20126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:47.746443 20120 server_base.cc:1034] running on GCE node
I20260928 11:01:47.747586 20120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:47.749192 20120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:47.750396 20120 hybrid_clock.cc:648] HybridClock initialized: now 1790593307750321 us; error 50 us; skew 500 ppm
I20260928 11:01:47.750761 20120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:47 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:47.754223 20120 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:01:47.755738 20120 webserver.cc:460] Webserver started at http://127.3.252.126:38575/ using document root <none> and password file <none>
I20260928 11:01:47.756460 20120 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:47.756570 20120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:47.762748 20120 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 11:01:47.765658 20136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:47.767066 20120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:01:47.767236 20120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "7b8001367dfc41ce947c16ab4a393483"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "411cc0a44e58eb72fa68f89bb0ce5ac0"
server_key_iv: "4e83e17e6dafc49d37e197fdf521c075"
server_key_version: "encryptionkey@0"
I20260928 11:01:47.768296 20120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:47.787405 20120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:47.791636 20120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:47.791913 20120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:47.809508 20120 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:37019
I20260928 11:01:47.809523 20187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:37019 every 8 connection(s)
I20260928 11:01:47.810971 20120 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:01:47.817135 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 20120
I20260928 11:01:47.817852 20188 sys_catalog.cc:263] Verifying existing consensus state
I20260928 11:01:47.817952 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:33585
--local_ip_for_outbound_sockets=127.3.252.65
--tserver_master_addrs=127.3.252.126:37019
--webserver_port=46509
--webserver_interface=127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:01:47.821116 20188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Bootstrap starting.
I20260928 11:01:47.845983 20188 log.cc:824] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:47.861472 20188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 11:01:47.862156 20188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Bootstrap complete.
I20260928 11:01:47.875370 20188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:47.876703 20188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8001367dfc41ce947c16ab4a393483, State: Initialized, Role: FOLLOWER
I20260928 11:01:47.877420 20188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:47.877707 20188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:01:47.877848 20188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:01:47.878038 20188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 1 FOLLOWER]: Advancing to term 2
I20260928 11:01:47.884042 20188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:47.884824 20188 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8001367dfc41ce947c16ab4a393483; no voters:
I20260928 11:01:47.885538 20188 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 11:01:47.885664 20192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 11:01:47.886831 20192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 LEADER]: Becoming Leader. State: Replica: 7b8001367dfc41ce947c16ab4a393483, State: Running, Role: LEADER
I20260928 11:01:47.887406 20192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:01:47.887794 20188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:01:47.889581 20193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7b8001367dfc41ce947c16ab4a393483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } } }
I20260928 11:01:47.889954 20193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:47.890406 20194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8001367dfc41ce947c16ab4a393483. Latest consensus state: current_term: 2 leader_uuid: "7b8001367dfc41ce947c16ab4a393483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } } }
I20260928 11:01:47.890692 20194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: This master's current role is: LEADER
I20260928 11:01:47.892905 20197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:01:47.903550 20197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:01:47.905011 20197 catalog_manager.cc:1259] Loaded cluster ID: 6cd9a27a3ed040f2b8e4206e76867738
I20260928 11:01:47.905169 20197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:01:47.909196 20197 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:01:47.910888 20197 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Loaded TSK: 0
I20260928 11:01:47.945608 20197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Generated new TSK 1
I20260928 11:01:47.946178 20197 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 11:01:48.176250 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:48.176549 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:48.176681 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:48.184556 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:48.184691 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:48.184883 20190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:01:48.195097 20190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:33585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=46509
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:48.196450 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:48.197750 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:48.212657 20215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:48.212677 20216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:48.212724 20218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:48.214927 20190 server_base.cc:1034] running on GCE node
I20260928 11:01:48.215575 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:48.216611 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:48.217905 20190 hybrid_clock.cc:648] HybridClock initialized: now 1790593308217830 us; error 54 us; skew 500 ppm
I20260928 11:01:48.218330 20190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:48 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593308, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:48.221679 20190 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:01:48.223604 20190 webserver.cc:460] Webserver started at http://127.3.252.65:46509/ using document root <none> and password file <none>
I20260928 11:01:48.224784 20190 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:48.225008 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:48.231314 20190 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 11:01:48.234304 20225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:48.235761 20190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:01:48.235930 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "5cdd07284bdb46d198bffda9961855cb"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "a67b097e9e10e29f54882e2a007ae381"
server_key_iv: "7ac4844cfc39d47f65409fe3222ed9ca"
server_key_version: "encryptionkey@0"
I20260928 11:01:48.236938 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:48.251555 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:48.256040 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:48.256292 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:48.257321 20190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:48.258736 20190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:48.258821 20190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:48.258924 20190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:48.259001 20190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:48.291251 20190 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:33585
I20260928 11:01:48.291308 20337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:33585 every 8 connection(s)
I20260928 11:01:48.292678 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:01:48.294482 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 20190
I20260928 11:01:48.295130 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:43971
--local_ip_for_outbound_sockets=127.3.252.66
--tserver_master_addrs=127.3.252.126:37019
--webserver_port=32985
--webserver_interface=127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 11:01:48 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593308, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:48.316761 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:48.295749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:50087 (local address 127.3.252.126:37019)
0928 11:01:48.296359 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:48.296377 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:48.297918 (+ 1541us) server_negotiation.cc:407] Connection header received
0928 11:01:48.299649 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:48.299662 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:48.300001 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:48.300311 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:48.302055 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.303271 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.304844 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.305495 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.305664 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:48.309536 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:48.309545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:48.309560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:48.309599 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:48.312433 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:48.313151 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:48.313158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:48.313163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:48.313315 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:48.313728 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:48.313731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:48.313733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:48.314238 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:48.314426 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:48.315188 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 11:01:48.315572 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260928 11:01:48.316759 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:48.296057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:50087)
0928 11:01:48.296922 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:48.296970 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:48.299110 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:48.300496 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:48.300506 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:48.301155 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:48.301871 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:48.301896 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.303455 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.303460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:48.304694 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:48.304714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.305722 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.305729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:48.306022 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:48.306663 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:48.306686 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:48.309247 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:48.312577 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:48.312587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:48.312603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:48.313012 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:48.313423 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:48.313428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:48.313432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:48.313614 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:48.314366 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:48.314375 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:48.314710 (+ 335us) client_negotiation.cc:769] Sending connection context
0928 11:01:48.314940 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 11:01:48.315497 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":347,"threads_started":1}
I20260928 11:01:48.319756 20338 heartbeater.cc:347] Connected to a master server at 127.3.252.126:37019
I20260928 11:01:48.320063 20338 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:48.320730 20338 heartbeater.cc:511] Master 127.3.252.126:37019 requested a full tablet report, sending...
I20260928 11:01:48.322990 20153 ts_manager.cc:194] Registered new tserver with Master: 5cdd07284bdb46d198bffda9961855cb (127.3.252.65:33585)
I20260928 11:01:48.325156 20153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:50087
W20260928 11:01:48.614851 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:48.615110 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:48.615211 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:48.622275 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:48.622390 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:48.622509 20342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:01:48.632077 20342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:43971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=32985
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:48.633270 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:48.634529 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:48.646760 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:48.646821 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:48.646730 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:48.647717 20342 server_base.cc:1034] running on GCE node
I20260928 11:01:48.648321 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:48.649217 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:48.650393 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790593308650344 us; error 39 us; skew 500 ppm
I20260928 11:01:48.650688 20342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:48 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593308, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:48.653576 20342 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:01:48.654978 20342 webserver.cc:460] Webserver started at http://127.3.252.66:32985/ using document root <none> and password file <none>
I20260928 11:01:48.655604 20342 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:48.655709 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:48.660758 20342 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 11:01:48.663317 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:48.664536 20342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 11:01:48.664673 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "e3c014f68f534cfb95da27367857b48b"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "493304c15623331923f20d02d43d2ad5"
server_key_iv: "2c926a45fae26c058dea5d31428f88c0"
server_key_version: "encryptionkey@0"
I20260928 11:01:48.665565 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:48.681213 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:48.685171 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:48.685420 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:48.686412 20342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:48.687645 20342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:48.687736 20342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:48.687866 20342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:48.687934 20342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:48.717046 20342 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:43971
I20260928 11:01:48.717137 20469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:43971 every 8 connection(s)
I20260928 11:01:48.718434 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:01:48.719231 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 20342
I20260928 11:01:48.719944 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:37811
--local_ip_for_outbound_sockets=127.3.252.67
--tserver_master_addrs=127.3.252.126:37019
--webserver_port=35943
--webserver_interface=127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 11:01:48 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593308, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:48.739069 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:48.721118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:39621 (local address 127.3.252.126:37019)
0928 11:01:48.721274 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:48.721278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:48.722623 (+ 1345us) server_negotiation.cc:407] Connection header received
0928 11:01:48.724411 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:48.724417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:48.724502 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:48.724590 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:48.726191 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.727094 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.728646 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.729246 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.729375 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:48.732848 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:48.732854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:48.732859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:48.732900 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:48.735835 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:48.736567 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:48.736572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:48.736575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:48.736716 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:48.737131 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:48.737136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:48.737139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:48.737520 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:48.737673 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:48.738451 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 11:01:48.738701 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 11:01:48.739802 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:48.721681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:39621)
0928 11:01:48.722451 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:48.722493 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:48.724142 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:48.724880 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:48.724890 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:48.725409 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:48.726027 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:48.726046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.727271 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.727276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:48.728508 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:48.728520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:48.729434 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:48.729438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:48.729622 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:48.730297 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:48.730319 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:48.732621 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:48.735983 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:48.735998 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:48.736013 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:48.736446 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:48.736810 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:48.736813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:48.736816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:48.737026 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:48.737660 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:48.737667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:48.738072 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 11:01:48.738299 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 11:01:48.738700 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":303,"threads_started":1}
I20260928 11:01:48.741257 20470 heartbeater.cc:347] Connected to a master server at 127.3.252.126:37019
I20260928 11:01:48.741609 20470 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:48.742352 20470 heartbeater.cc:511] Master 127.3.252.126:37019 requested a full tablet report, sending...
I20260928 11:01:48.743762 20153 ts_manager.cc:194] Registered new tserver with Master: e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
I20260928 11:01:48.744563 20153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:39621
W20260928 11:01:49.046449 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:01:49.046743 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:01:49.046890 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:01:49.054173 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:01:49.054288 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:01:49.054438 20473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:01:49.064240 20473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:37811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=35943
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:01:49.065506 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:01:49.066810 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:01:49.079577 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:49.079627 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:01:49.079653 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:01:49.080730 20473 server_base.cc:1034] running on GCE node
I20260928 11:01:49.081318 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:01:49.082357 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:01:49.083590 20473 hybrid_clock.cc:648] HybridClock initialized: now 1790593309083530 us; error 52 us; skew 500 ppm
I20260928 11:01:49.083904 20473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:01:49 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:01:49.087059 20473 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:01:49.088624 20473 webserver.cc:460] Webserver started at http://127.3.252.67:35943/ using document root <none> and password file <none>
I20260928 11:01:49.089339 20473 fs_manager.cc:359] Metadata directory not provided
I20260928 11:01:49.089479 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:01:49.095285 20473 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 11:01:49.098263 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:49.099646 20473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:01:49.099807 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "cefbcea617ea4016a72abd5a8df9e859"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "744e19e566c7f2501a195ab97bec596d"
server_key_iv: "4bb3db4b638d6a7dd26c078864e69fa8"
server_key_version: "encryptionkey@0"
I20260928 11:01:49.100843 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:01:49.112597 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:01:49.116461 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:01:49.116720 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:01:49.117653 20473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:01:49.118952 20473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:01:49.119067 20473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:49.119199 20473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:01:49.119261 20473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:01:49.149179 20473 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:37811
I20260928 11:01:49.149233 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:37811 every 8 connection(s)
I20260928 11:01:49.150640 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:01:49.154304 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 20473
Sep 28 11:01:49 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:01:49.170162 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.155229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.3.252.126:37019)
0928 11:01:49.155653 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.155659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.155978 (+ 319us) server_negotiation.cc:407] Connection header received
0928 11:01:49.156135 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.156140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.156219 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.156335 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:49.158005 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.158919 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.160398 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.161079 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.161206 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.164107 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:49.164114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:49.164119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:49.164156 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:49.167437 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.168155 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.168161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.168165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.168302 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.168700 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.168703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.168705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.168977 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:49.169153 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.169527 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.169751 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":226,"threads_started":1}
I20260928 11:01:49.171825 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.153004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:55263 (local address 127.3.252.126:37019)
0928 11:01:49.153219 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.153226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.154429 (+ 1203us) server_negotiation.cc:407] Connection header received
0928 11:01:49.156648 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.156655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.156725 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.156822 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:49.158469 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.159339 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.161228 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.161880 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.162008 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.165596 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:49.165603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:49.165608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:49.165656 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:49.168480 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.169246 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.169250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.169252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.169350 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.169958 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.169963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.169967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.170288 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:49.170426 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.171278 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.171466 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 11:01:49.172928 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.153482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:55263)
0928 11:01:49.154264 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.154316 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:49.156360 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:49.157033 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:49.157041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:49.157576 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:01:49.158298 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.158321 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.159509 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.159513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.161052 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.161067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.162108 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.162116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.162363 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.162958 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:01:49.162981 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:01:49.165362 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:01:49.168617 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:49.168626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:49.168645 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:49.169117 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:49.169502 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:01:49.169507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:01:49.169510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:01:49.169812 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:01:49.170494 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:01:49.170500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:01:49.170871 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 11:01:49.171156 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 11:01:49.171662 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":281,"threads_started":1}
I20260928 11:01:49.174263 20601 heartbeater.cc:347] Connected to a master server at 127.3.252.126:37019
I20260928 11:01:49.174530 20601 heartbeater.cc:464] Registering TS with master...
I20260928 11:01:49.175215 20601 heartbeater.cc:511] Master 127.3.252.126:37019 requested a full tablet report, sending...
I20260928 11:01:49.176698 20153 ts_manager.cc:194] Registered new tserver with Master: cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811)
I20260928 11:01:49.177539 20153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:55263
I20260928 11:01:49.184911 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:01:49.191249 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.185895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.3.252.126:37019)
0928 11:01:49.186056 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.186060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.186388 (+ 328us) server_negotiation.cc:407] Connection header received
0928 11:01:49.186577 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.186580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.186723 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.186843 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:01:49.187197 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.188024 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.189501 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.189962 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.190064 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.190541 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:01:49.190755 (+ 214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 11:01:49.191058 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43766: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 11:01:49.191363 20340 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43766: token expired
W20260928 11:01:49.191900 20076 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.252.126:37019)
I20260928 11:01:49.203982 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.192319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.3.252.126:37019)
0928 11:01:49.192449 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.192453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.192469 (+ 16us) server_negotiation.cc:407] Connection header received
0928 11:01:49.192592 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.192595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.192649 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.192721 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:49.194103 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.194826 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.196187 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.196549 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.196632 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.199212 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:49.199217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:49.199220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:49.199251 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:49.201707 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.202397 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.202401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.202403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.202518 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.202791 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.202794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.202796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.203103 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:49.203279 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.203523 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.203690 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 11:01:49.207612 20153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:01:49.214726 20153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:01:49.248883 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.240169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.3.252.66:43971)
0928 11:01:49.241311 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.241318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.242312 (+ 994us) server_negotiation.cc:407] Connection header received
0928 11:01:49.242404 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.242411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.242634 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.242808 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:49.243543 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.244552 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.246767 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.247759 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.247896 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.248112 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.248397 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.248569 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":917,"threads_started":1}
I20260928 11:01:49.249293 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.226586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.3.252.67:37811)
0928 11:01:49.238812 (+ 12226us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.238821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.239577 (+ 756us) server_negotiation.cc:407] Connection header received
0928 11:01:49.240006 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.240015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.240265 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.240453 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:49.242035 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.243564 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.246354 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.247843 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.247954 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.248187 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.248795 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.248958 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12048,"thread_start_us":11871,"threads_started":1}
I20260928 11:01:49.249480 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.226703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51670)
0928 11:01:49.239418 (+ 12715us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.239459 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:49.239748 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:49.240877 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:49.240885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:49.240930 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:49.241455 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.241469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.243401 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.243405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.246196 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.246206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.248336 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.248345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.248529 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.248621 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 11:01:49.248706 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 11:01:49.248833 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12552,"mutex_wait_us":153,"spinlock_wait_cycles":9856,"thread_start_us":12233,"threads_started":1}
I20260928 11:01:49.249536 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.226966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.3.252.65:33585)
0928 11:01:49.238059 (+ 11093us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.238069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.239579 (+ 1510us) server_negotiation.cc:407] Connection header received
0928 11:01:49.240008 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.240016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.240273 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.240460 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:49.241668 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.243277 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.246371 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.247843 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.247993 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.248153 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.249072 (+ 919us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.249238 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10843,"thread_start_us":11072,"threads_started":1}
I20260928 11:01:49.248980 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.239893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:43971 (local address 127.0.0.1:48612)
0928 11:01:49.242078 (+ 2185us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.242096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:49.242205 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:49.243009 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:49.243013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:49.243036 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:49.243381 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.243393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.244690 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.244695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.246622 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.246635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.247975 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.247981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.248172 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.248222 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 11:01:49.248359 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 11:01:49.248569 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2054,"mutex_wait_us":1,"thread_start_us":1884,"threads_started":1}
I20260928 11:01:49.250226 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.226278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:37811 (local address 127.0.0.1:40890)
0928 11:01:49.239418 (+ 13140us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.239458 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:49.239748 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:49.240876 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:49.240884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:49.240935 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:49.241720 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.241738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.243711 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.243716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.246169 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.246189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.248372 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.248377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.248546 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.248626 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 11:01:49.249710 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0928 11:01:49.249868 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12844,"mutex_wait_us":505,"spinlock_wait_cycles":25600,"thread_start_us":12827,"threads_started":1}
I20260928 11:01:49.253319 20273 tablet_service.cc:1467] Processing CreateTablet for tablet 10faf6f64bec4268b3aafed2f9b39e2c (DEFAULT_TABLE table=test-table [id=20045541baef4b9ca05210dd75f9e17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:49.253682 20536 tablet_service.cc:1467] Processing CreateTablet for tablet 10faf6f64bec4268b3aafed2f9b39e2c (DEFAULT_TABLE table=test-table [id=20045541baef4b9ca05210dd75f9e17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:49.253682 20405 tablet_service.cc:1467] Processing CreateTablet for tablet 10faf6f64bec4268b3aafed2f9b39e2c (DEFAULT_TABLE table=test-table [id=20045541baef4b9ca05210dd75f9e17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:01:49.254701 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10faf6f64bec4268b3aafed2f9b39e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:49.255023 20405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10faf6f64bec4268b3aafed2f9b39e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:49.255017 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10faf6f64bec4268b3aafed2f9b39e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:01:49.267153 20617 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Bootstrap starting.
I20260928 11:01:49.270118 20618 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Bootstrap starting.
I20260928 11:01:49.271270 20617 tablet_bootstrap.cc:654] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:49.271435 20619 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Bootstrap starting.
I20260928 11:01:49.272704 20617 log.cc:824] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:49.273713 20618 tablet_bootstrap.cc:654] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:49.275139 20618 log.cc:824] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:49.275353 20619 tablet_bootstrap.cc:654] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Neither blocks nor log segments found. Creating new log.
I20260928 11:01:49.276119 20617 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: No bootstrap required, opened a new log
I20260928 11:01:49.276434 20617 ts_tablet_manager.cc:1397] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260928 11:01:49.276415 20619 log.cc:824] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Log is configured to *not* fsync() on all Append() calls
I20260928 11:01:49.278189 20618 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: No bootstrap required, opened a new log
I20260928 11:01:49.278472 20618 ts_tablet_manager.cc:1397] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260928 11:01:49.279418 20619 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: No bootstrap required, opened a new log
I20260928 11:01:49.279676 20619 ts_tablet_manager.cc:1397] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 11:01:49.284317 20617 raft_consensus.cc:348] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.284706 20617 raft_consensus.cc:374] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:49.284818 20617 raft_consensus.cc:729] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdd07284bdb46d198bffda9961855cb, State: Initialized, Role: FOLLOWER
I20260928 11:01:49.285413 20617 consensus_queue.cc:260] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.286579 20338 heartbeater.cc:503] Master 127.3.252.126:37019 was elected leader, sending a full tablet report...
I20260928 11:01:49.287081 20617 ts_tablet_manager.cc:1428] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260928 11:01:49.288836 20618 raft_consensus.cc:348] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.289371 20618 raft_consensus.cc:374] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:49.289476 20618 raft_consensus.cc:729] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c014f68f534cfb95da27367857b48b, State: Initialized, Role: FOLLOWER
I20260928 11:01:49.290179 20618 consensus_queue.cc:260] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.294076 20619 raft_consensus.cc:348] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.294544 20619 raft_consensus.cc:374] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:01:49.294561 20470 heartbeater.cc:503] Master 127.3.252.126:37019 was elected leader, sending a full tablet report...
I20260928 11:01:49.294675 20619 raft_consensus.cc:729] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cefbcea617ea4016a72abd5a8df9e859, State: Initialized, Role: FOLLOWER
W20260928 11:01:49.294818 20339 tablet.cc:2369] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:01:49.295291 20619 consensus_queue.cc:260] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.296281 20618 ts_tablet_manager.cc:1428] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Time spent starting tablet: real 0.018s user 0.006s sys 0.010s
I20260928 11:01:49.298257 20601 heartbeater.cc:503] Master 127.3.252.126:37019 was elected leader, sending a full tablet report...
I20260928 11:01:49.298835 20619 ts_tablet_manager.cc:1428] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Time spent starting tablet: real 0.019s user 0.009s sys 0.007s
I20260928 11:01:49.314851 20624 raft_consensus.cc:482] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:01:49.315162 20624 raft_consensus.cc:504] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.316783 20624 leader_election.cc:288] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), 5cdd07284bdb46d198bffda9961855cb (127.3.252.65:33585)
I20260928 11:01:49.323769 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.317167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:60701 (local address 127.3.252.67:37811)
0928 11:01:49.317385 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.317391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.318386 (+ 995us) server_negotiation.cc:407] Connection header received
0928 11:01:49.318529 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.318533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.318611 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.318705 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:49.319333 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.320041 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.321894 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.322843 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.322988 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.323061 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.323399 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.323523 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 11:01:49.323794 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.317345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:34631 (local address 127.3.252.65:33585)
0928 11:01:49.317543 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.317549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.318180 (+ 631us) server_negotiation.cc:407] Connection header received
0928 11:01:49.318388 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.318393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.318466 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.318546 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:01:49.319134 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.319880 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.321980 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.322888 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.323025 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.323099 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.323426 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.323556 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 11:01:49.323928 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.317272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:37811 (local address 127.3.252.66:60701)
0928 11:01:49.318280 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.318293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:49.318420 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:49.318727 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:49.318731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:49.318764 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:49.319188 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.319201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.320168 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.320172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.321708 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.321723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.323070 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.323078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.323231 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.323275 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 11:01:49.323362 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 11:01:49.323490 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":47,"thread_start_us":697,"threads_started":1}
I20260928 11:01:49.323969 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.317230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.3.252.66:34631)
0928 11:01:49.318049 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.318076 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 11:01:49.318247 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:01:49.318583 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:01:49.318588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:01:49.318620 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:01:49.318995 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.319005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.320008 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.320014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.321746 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:01:49.321759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.323105 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.323112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:01:49.323242 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.323307 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 11:01:49.323386 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 11:01:49.323509 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":65,"thread_start_us":427,"threads_started":1}
I20260928 11:01:49.325153 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10faf6f64bec4268b3aafed2f9b39e2c" candidate_uuid: "e3c014f68f534cfb95da27367857b48b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cefbcea617ea4016a72abd5a8df9e859" is_pre_election: true
I20260928 11:01:49.325153 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10faf6f64bec4268b3aafed2f9b39e2c" candidate_uuid: "e3c014f68f534cfb95da27367857b48b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdd07284bdb46d198bffda9961855cb" is_pre_election: true
I20260928 11:01:49.325726 20293 raft_consensus.cc:2454] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c014f68f534cfb95da27367857b48b in term 0.
I20260928 11:01:49.325726 20556 raft_consensus.cc:2454] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c014f68f534cfb95da27367857b48b in term 0.
I20260928 11:01:49.326509 20359 leader_election.cc:302] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cefbcea617ea4016a72abd5a8df9e859, e3c014f68f534cfb95da27367857b48b; no voters:
I20260928 11:01:49.326999 20624 raft_consensus.cc:2781] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:01:49.327138 20624 raft_consensus.cc:482] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:01:49.327236 20624 raft_consensus.cc:3037] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:49.329346 20624 raft_consensus.cc:504] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.329968 20624 leader_election.cc:288] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [CANDIDATE]: Term 1 election: Requested vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), 5cdd07284bdb46d198bffda9961855cb (127.3.252.65:33585)
I20260928 11:01:49.330544 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10faf6f64bec4268b3aafed2f9b39e2c" candidate_uuid: "e3c014f68f534cfb95da27367857b48b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cefbcea617ea4016a72abd5a8df9e859"
I20260928 11:01:49.330546 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10faf6f64bec4268b3aafed2f9b39e2c" candidate_uuid: "e3c014f68f534cfb95da27367857b48b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdd07284bdb46d198bffda9961855cb"
I20260928 11:01:49.330763 20556 raft_consensus.cc:3037] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:49.330763 20293 raft_consensus.cc:3037] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:01:49.334100 20293 raft_consensus.cc:2454] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c014f68f534cfb95da27367857b48b in term 1.
I20260928 11:01:49.334100 20556 raft_consensus.cc:2454] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c014f68f534cfb95da27367857b48b in term 1.
I20260928 11:01:49.334537 20359 leader_election.cc:302] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb, cefbcea617ea4016a72abd5a8df9e859, e3c014f68f534cfb95da27367857b48b; no voters:
I20260928 11:01:49.334877 20624 raft_consensus.cc:2781] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:01:49.335362 20624 raft_consensus.cc:686] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 1 LEADER]: Becoming Leader. State: Replica: e3c014f68f534cfb95da27367857b48b, State: Running, Role: LEADER
I20260928 11:01:49.335856 20624 consensus_queue.cc:237] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:01:49.340286 20153 catalog_manager.cc:5495] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c014f68f534cfb95da27367857b48b (127.3.252.66). New cstate: current_term: 1 leader_uuid: "e3c014f68f534cfb95da27367857b48b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } health_report { overall_health: UNKNOWN } } }
I20260928 11:01:49.378115 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.372868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.3.252.66:43971)
0928 11:01:49.373079 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.373086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.373191 (+ 105us) server_negotiation.cc:407] Connection header received
0928 11:01:49.373371 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.373375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.373453 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.373575 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:01:49.373896 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.374599 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.375949 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.376390 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.376478 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.376922 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:01:49.377442 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:01:49.377544 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.377772 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.377887 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 11:01:49.386984 20293 raft_consensus.cc:1261] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Refusing update from remote peer e3c014f68f534cfb95da27367857b48b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:49.386974 20556 raft_consensus.cc:1261] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Refusing update from remote peer e3c014f68f534cfb95da27367857b48b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:01:49.388108 20628 consensus_queue.cc:1035] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:49.388474 20624 consensus_queue.cc:1035] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:01:49.403323 20633 mvcc.cc:205] Tried to move back new op lower bound from 7334270195237257216 to 7334270195043459072. Current Snapshot: MvccSnapshot[applied={T|T < 7334270195237257216}]
I20260928 11:01:49.405969 20632 mvcc.cc:205] Tried to move back new op lower bound from 7334270195237257216 to 7334270195043459072. Current Snapshot: MvccSnapshot[applied={T|T < 7334270195237257216 or (T in {7334270195237257216})}]
I20260928 11:01:49.412168 20153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:49.412436 20153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:01:49.416059 20153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Sending DeleteTablet for 3 replicas of tablet 10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:01:49.417169 20536 tablet_service.cc:1514] Processing DeleteTablet for tablet 10faf6f64bec4268b3aafed2f9b39e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:49 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40890
I20260928 11:01:49.417347 20273 tablet_service.cc:1514] Processing DeleteTablet for tablet 10faf6f64bec4268b3aafed2f9b39e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:49 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:51670
I20260928 11:01:49.417430 20405 tablet_service.cc:1514] Processing DeleteTablet for tablet 10faf6f64bec4268b3aafed2f9b39e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:01:49 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:48612
I20260928 11:01:49.418497 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 20190
I20260928 11:01:49.430199 20642 tablet_replica.cc:330] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: stopping tablet replica
I20260928 11:01:49.430833 20642 raft_consensus.cc:2229] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:01:49.431633 20642 raft_consensus.cc:2258] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:49.433982 20640 tablet_replica.cc:330] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: stopping tablet replica
I20260928 11:01:49.434116 20642 ts_tablet_manager.cc:1905] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:49.434408 20640 raft_consensus.cc:2229] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:49.434899 20640 raft_consensus.cc:2258] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:49.435168 20641 tablet_replica.cc:330] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: stopping tablet replica
I20260928 11:01:49.435532 20641 raft_consensus.cc:2229] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:01:49.436900 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.419304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.3.252.65:33585)
0928 11:01:49.419479 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.419487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.419630 (+ 143us) server_negotiation.cc:407] Connection header received
0928 11:01:49.419819 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.419824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.419945 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.420058 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:49.421805 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.423064 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.425031 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.425772 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.425945 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.429563 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:49.429583 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:49.429607 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:49.429659 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:49.433450 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.434274 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.434283 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.434286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.434442 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.434859 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.434864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.434867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.435731 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:49.436010 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.436321 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.436554 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 11:01:49.437093 20641 raft_consensus.cc:2258] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:01:49.437551 20333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:49.438762 20640 ts_tablet_manager.cc:1905] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:01:49.445868 20640 ts_tablet_manager.cc:1918] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:49.446029 20640 log.cc:1196] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:01:49.446528 20640 ts_tablet_manager.cc:1939] T 10faf6f64bec4268b3aafed2f9b39e2c P cefbcea617ea4016a72abd5a8df9e859: Deleting consensus metadata
I20260928 11:01:49.448395 20642 ts_tablet_manager.cc:1918] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:01:49.448381 20138 catalog_manager.cc:4841] TS cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811): tablet 10faf6f64bec4268b3aafed2f9b39e2c (table test-table [id=20045541baef4b9ca05210dd75f9e17c]) successfully deleted
I20260928 11:01:49.448577 20642 log.cc:1196] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:01:49.449079 20642 ts_tablet_manager.cc:1939] T 10faf6f64bec4268b3aafed2f9b39e2c P e3c014f68f534cfb95da27367857b48b: Deleting consensus metadata
I20260928 11:01:49.451007 20138 catalog_manager.cc:4841] TS e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971): tablet 10faf6f64bec4268b3aafed2f9b39e2c (table test-table [id=20045541baef4b9ca05210dd75f9e17c]) successfully deleted
I20260928 11:01:49.537210 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 20190
I20260928 11:01:49.552284 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 20342
W20260928 11:01:49.552173 20137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.252.65:33585 (error 108)
I20260928 11:01:49.554994 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.554526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51688)
0928 11:01:49.554704 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.554821 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 11:01:49.555546 20137 catalog_manager.cc:4584] TS 5cdd07284bdb46d198bffda9961855cb (127.3.252.65:33585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10faf6f64bec4268b3aafed2f9b39e2c: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
I20260928 11:01:49.570075 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.553093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.3.252.66:43971)
0928 11:01:49.553291 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.553298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.553357 (+ 59us) server_negotiation.cc:407] Connection header received
0928 11:01:49.553566 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.553572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.553663 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.553762 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:49.555432 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.556707 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.558346 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.559002 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.559120 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.562417 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:49.562435 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:49.562455 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:49.562543 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:49.566690 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.567586 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.567596 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.567600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.567777 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.568187 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.568192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.568196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.568726 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:49.569042 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.569424 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.569662 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 11:01:49.570797 20465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:49.609221 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.608743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51690)
0928 11:01:49.608955 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.609066 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 11:01:49.610235 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.609930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51702)
0928 11:01:49.610051 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.610133 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 11:01:49.670889 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.670259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51714)
0928 11:01:49.670569 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.670701 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 11:01:49.671932 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.671583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51722)
0928 11:01:49.671724 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.671805 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 11:01:49.681859 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 20342
I20260928 11:01:49.697011 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 20473
I20260928 11:01:49.710234 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.697715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.3.252.67:37811)
0928 11:01:49.697979 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 11:01:49.697985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:01:49.698005 (+ 20us) server_negotiation.cc:407] Connection header received
0928 11:01:49.698099 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:01:49.698103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:01:49.698175 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:01:49.698258 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:01:49.699644 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.700351 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.701853 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:01:49.702252 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:01:49.702373 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:01:49.704834 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:01:49.704848 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:01:49.704871 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:01:49.704905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:01:49.707531 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.708195 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.708202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.708205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.708350 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:01:49.708705 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:01:49.708708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:01:49.708710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:01:49.709095 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:01:49.709362 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 11:01:49.709688 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 11:01:49.709920 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 11:01:49.710808 20596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:49.752449 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.751992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51726)
0928 11:01:49.752194 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.752285 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 11:01:49.753459 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:01:49.753083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:33585 (local address 127.0.0.1:51738)
0928 11:01:49.753277 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 11:01:49.753349 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 11:01:49.798763 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 20473
I20260928 11:01:49.812489 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 20120
I20260928 11:01:49.813658 20183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:01:49.925199 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 20120
I20260928 11:02:09.943637 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:37019
--webserver_interface=127.3.252.126
--webserver_port=38575
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:10.268653 20648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:10.268962 20648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:10.269068 20648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:10.276340 20648 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 11:02:10.276443 20648 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 11:02:10.276497 20648 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 11:02:10.276564 20648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:10.276639 20648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:10.276737 20648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:10.276801 20648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:10.276872 20648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:10.286613 20648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=38575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:10.287871 20648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:10.289143 20648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:10.300889 20656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:10.300999 20653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:10.300966 20654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:10.301470 20648 server_base.cc:1034] running on GCE node
I20260928 11:02:10.302294 20648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:10.303649 20648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:10.304872 20648 hybrid_clock.cc:648] HybridClock initialized: now 1790593330304799 us; error 36 us; skew 500 ppm
I20260928 11:02:10.305189 20648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:10 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:10.308254 20648 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:02:10.309621 20648 webserver.cc:460] Webserver started at http://127.3.252.126:38575/ using document root <none> and password file <none>
I20260928 11:02:10.310304 20648 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:10.310405 20648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:10.315739 20648 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 11:02:10.318472 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:10.319713 20648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:10.319883 20648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "7b8001367dfc41ce947c16ab4a393483"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "411cc0a44e58eb72fa68f89bb0ce5ac0"
server_key_iv: "4e83e17e6dafc49d37e197fdf521c075"
server_key_version: "encryptionkey@0"
I20260928 11:02:10.320797 20648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:10.351526 20648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:10.356104 20648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:10.356400 20648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:10.374891 20648 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:37019
I20260928 11:02:10.374908 20714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:37019 every 8 connection(s)
I20260928 11:02:10.376461 20648 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:10.383311 20715 sys_catalog.cc:263] Verifying existing consensus state
I20260928 11:02:10.386107 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 20648
I20260928 11:02:10.386993 20715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Bootstrap starting.
I20260928 11:02:10.387128 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:33585
--local_ip_for_outbound_sockets=127.3.252.65
--tserver_master_addrs=127.3.252.126:37019
--webserver_port=46509
--webserver_interface=127.3.252.65
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:02:10.409818 20715 log.cc:824] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:10.426366 20715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 11:02:10.426913 20715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Bootstrap complete.
I20260928 11:02:10.436260 20715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:02:10.437180 20715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8001367dfc41ce947c16ab4a393483, State: Initialized, Role: FOLLOWER
I20260928 11:02:10.437812 20715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:02:10.438088 20715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:10.438199 20715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:10.438375 20715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 2 FOLLOWER]: Advancing to term 3
I20260928 11:02:10.442359 20715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:02:10.442873 20715 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8001367dfc41ce947c16ab4a393483; no voters:
I20260928 11:02:10.443533 20715 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 11:02:10.443676 20719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 11:02:10.444774 20719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [term 3 LEADER]: Becoming Leader. State: Replica: 7b8001367dfc41ce947c16ab4a393483, State: Running, Role: LEADER
I20260928 11:02:10.445387 20719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } }
I20260928 11:02:10.445698 20715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:10.447886 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7b8001367dfc41ce947c16ab4a393483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } } }
I20260928 11:02:10.448222 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:10.453225 20721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8001367dfc41ce947c16ab4a393483. Latest consensus state: current_term: 3 leader_uuid: "7b8001367dfc41ce947c16ab4a393483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8001367dfc41ce947c16ab4a393483" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 37019 } } }
I20260928 11:02:10.453686 20721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:10.455184 20724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:10.464663 20724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:10.466434 20724 catalog_manager.cc:1259] Loaded cluster ID: 6cd9a27a3ed040f2b8e4206e76867738
I20260928 11:02:10.466620 20724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:10.470999 20724 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:10.472920 20724 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Loaded TSK: 0
I20260928 11:02:10.473189 20724 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7b8001367dfc41ce947c16ab4a393483: Loaded TSK: 1
F20260928 11:02:10.474500 20724 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f232d82b9ad google::LogMessage::Fail() at ??:0
@ 0x7f232d82e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f232d82b4de google::LogMessage::Flush() at ??:0
@ 0x7f232d82ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f233158f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f233157e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f234325a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2343259783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f23432e52a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f23432e50c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f234332f5e9 std::function<>::operator()() at ??:0
@ 0x7f234325dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f234325bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f23432e3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f23432e3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2330477345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f233047a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f233047a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2330455eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f233372f6db start_thread at ??:0
@ 0x7f232c6c561f clone at ??:0
W20260928 11:02:10.766649 20717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:10.766950 20717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:10.767072 20717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:10.774458 20717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:10.774571 20717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:10.774711 20717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:02:10.784646 20717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:33585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=46509
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:10.785965 20717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:10.787278 20717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:10.799856 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:10.800050 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:10.800100 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:10.800953 20717 server_base.cc:1034] running on GCE node
I20260928 11:02:10.801581 20717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:10.802632 20717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:10.803844 20717 hybrid_clock.cc:648] HybridClock initialized: now 1790593330803780 us; error 58 us; skew 500 ppm
I20260928 11:02:10.804169 20717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:10 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:10.807235 20717 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:02:10.808630 20717 webserver.cc:460] Webserver started at http://127.3.252.65:46509/ using document root <none> and password file <none>
I20260928 11:02:10.809286 20717 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:10.809402 20717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:10.814795 20717 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 11:02:10.817569 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:10.818886 20717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:02:10.819031 20717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "5cdd07284bdb46d198bffda9961855cb"
format_stamp: "Formatted at 2026-09-28 11:01:25 on dist-test-slave-djjw"
server_key: "a67b097e9e10e29f54882e2a007ae381"
server_key_iv: "7ac4844cfc39d47f65409fe3222ed9ca"
server_key_version: "encryptionkey@0"
I20260928 11:02:10.819949 20717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:10.844635 20717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:10.849032 20717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:10.849288 20717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:10.850337 20717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:10.852174 20779 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 11:02:10.857885 20717 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 11:02:10.858001 20717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 11:02:10.858115 20717 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 11:02:10.861150 20717 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 11:02:10.861230 20717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260928 11:02:10.861596 20779 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Bootstrap starting.
I20260928 11:02:10.869716 20779 log.cc:824] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:10.883968 20779 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 11:02:10.884594 20779 tablet_bootstrap.cc:492] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Bootstrap complete.
I20260928 11:02:10.885035 20779 ts_tablet_manager.cc:1397] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.011s
I20260928 11:02:10.894450 20779 raft_consensus.cc:348] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:02:10.895450 20779 raft_consensus.cc:729] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdd07284bdb46d198bffda9961855cb, State: Initialized, Role: FOLLOWER
I20260928 11:02:10.896128 20779 consensus_queue.cc:260] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:02:10.897657 20779 ts_tablet_manager.cc:1428] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260928 11:02:10.900522 20717 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:33585
I20260928 11:02:10.900554 20887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:33585 every 8 connection(s)
I20260928 11:02:10.901647 20717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:02:10.904749 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 20717
I20260928 11:02:10.905480 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:43971
--local_ip_for_outbound_sockets=127.3.252.66
--tserver_master_addrs=127.3.252.126:37019
--webserver_port=32985
--webserver_interface=127.3.252.66
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:02:10.906143 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:10.904142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:43057)
0928 11:02:10.905006 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:10.905226 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
W20260928 11:02:10.906714 20888 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:37019 (0 consecutive failures): Network error: Failed to ping master at 127.3.252.126:37019: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
I20260928 11:02:10.907493 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:10.907190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:48525)
0928 11:02:10.907331 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:10.907399 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 11:02:10.908253 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:10.907940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:59459)
0928 11:02:10.908075 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:10.908144 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 11:02:10.908455 20888 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 11:02:11.220840 20891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:11.221141 20891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:11.221239 20891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:11.228860 20891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:11.228974 20891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:11.229094 20891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:02:11.238709 20891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:43971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=32985
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:11.239945 20891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:11.241151 20891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:11.253301 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:11.253274 20897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:11.253733 20896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:11.254073 20891 server_base.cc:1034] running on GCE node
I20260928 11:02:11.254588 20891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:11.255411 20891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:11.256595 20891 hybrid_clock.cc:648] HybridClock initialized: now 1790593331256547 us; error 39 us; skew 500 ppm
I20260928 11:02:11.256871 20891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:11 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593331, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:11.259774 20891 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:02:11.261158 20891 webserver.cc:460] Webserver started at http://127.3.252.66:32985/ using document root <none> and password file <none>
I20260928 11:02:11.261792 20891 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:11.261955 20891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:11.267045 20891 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:02:11.269698 20906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:11.271028 20891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:11.271174 20891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "e3c014f68f534cfb95da27367857b48b"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "493304c15623331923f20d02d43d2ad5"
server_key_iv: "2c926a45fae26c058dea5d31428f88c0"
server_key_version: "encryptionkey@0"
I20260928 11:02:11.272119 20891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:11.293058 20891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:11.297189 20891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:11.297438 20891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:11.298454 20891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:11.299793 20891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:11.299876 20891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:11.299993 20891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:11.300060 20891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:11.329866 20891 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:43971
I20260928 11:02:11.329936 21018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:43971 every 8 connection(s)
I20260928 11:02:11.331238 20891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:02:11.335844 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:11.334057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:40959)
0928 11:02:11.334730 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:11.334986 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":250,"threads_started":1}
W20260928 11:02:11.336529 21019 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:37019 (0 consecutive failures): Network error: Failed to ping master at 127.3.252.126:37019: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
I20260928 11:02:11.337263 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 20891
I20260928 11:02:11.337287 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:11.336981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:33643)
0928 11:02:11.337123 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:11.337188 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 11:02:11.337935 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:37811
--local_ip_for_outbound_sockets=127.3.252.67
--tserver_master_addrs=127.3.252.126:37019
--webserver_port=35943
--webserver_interface=127.3.252.67
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:02:11.338179 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:11.337808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:41995)
0928 11:02:11.337966 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:11.338044 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 11:02:11.338405 21019 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 11:02:11.657178 21022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:11.657471 21022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:11.657564 21022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:11.664791 21022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:11.664909 21022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:11.665042 21022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:02:11.674736 21022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:37811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=35943
--tserver_master_addrs=127.3.252.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:11.676023 21022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:11.677414 21022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:11.690073 21030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:11.690104 21028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:11.690135 21027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:11.690766 21022 server_base.cc:1034] running on GCE node
I20260928 11:02:11.691289 21022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:11.692225 21022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:11.693434 21022 hybrid_clock.cc:648] HybridClock initialized: now 1790593331693382 us; error 45 us; skew 500 ppm
I20260928 11:02:11.693744 21022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:11 dist-test-slave-djjw krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593331, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:11.696755 21022 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:02:11.698195 21022 webserver.cc:460] Webserver started at http://127.3.252.67:35943/ using document root <none> and password file <none>
I20260928 11:02:11.698920 21022 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:11.699033 21022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:11.704031 21022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 11:02:11.706604 21037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:11.707829 21022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:11.707979 21022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "cefbcea617ea4016a72abd5a8df9e859"
format_stamp: "Formatted at 2026-09-28 11:01:26 on dist-test-slave-djjw"
server_key: "744e19e566c7f2501a195ab97bec596d"
server_key_iv: "4bb3db4b638d6a7dd26c078864e69fa8"
server_key_version: "encryptionkey@0"
I20260928 11:02:11.708949 21022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:11.723539 21022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:11.727612 21022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:11.727855 21022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:11.728793 21022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:11.730162 21022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:11.730255 21022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:11.730357 21022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:11.730434 21022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:11.759879 21022 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:37811
I20260928 11:02:11.759943 21149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:37811 every 8 connection(s)
I20260928 11:02:11.761253 21022 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:02:11.765933 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:11.764117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:56389)
0928 11:02:11.764808 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:11.765047 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":255,"threads_started":1}
W20260928 11:02:11.766628 21150 heartbeater.cc:644] Failed to heartbeat to 127.3.252.126:37019 (0 consecutive failures): Network error: Failed to ping master at 127.3.252.126:37019: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
I20260928 11:02:11.767426 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:11.767080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:35825)
0928 11:02:11.767239 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:11.767313 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 11:02:11.768172 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:11.767885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:38777)
0928 11:02:11.767999 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:11.768071 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 11:02:11.768386 21150 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 11:02:11.770129 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21022
I20260928 11:02:11.909698 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:11.908973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:54615)
0928 11:02:11.909392 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:11.909498 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":188,"threads_started":1}
I20260928 11:02:12.339916 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:12.339055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:55979)
0928 11:02:12.339562 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:12.339672 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
I20260928 11:02:12.626857 21156 raft_consensus.cc:482] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:12.627139 21156 raft_consensus.cc:504] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:02:12.628645 21156 leader_election.cc:288] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
Sep 28 11:02:12 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
Sep 28 11:02:12 dist-test-slave-djjw krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 11:02:12.653143 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:12.629778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:43971 (local address 127.3.252.65:47991)
0928 11:02:12.632057 (+ 2279us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:12.632072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:12.632193 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:12.634331 (+ 2138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:12.634335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:12.634946 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:12.635483 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:12.635502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.636963 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.636967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:12.638334 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:12.638346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.639231 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.639237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:12.639502 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:12.643597 (+ 4095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:12.643614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:12.646259 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:12.649357 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:12.649362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:12.649366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:12.649901 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:12.650285 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:12.650290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:12.650293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:12.650574 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:12.651629 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:12.651632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:12.651769 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 11:02:12.651988 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 11:02:12.652699 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2177,"mutex_wait_us":281,"spinlock_wait_cycles":61056,"thread_start_us":2004,"threads_started":1}
I20260928 11:02:12.653297 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:12.629017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:37811 (local address 127.3.252.65:52349)
0928 11:02:12.629568 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:12.629615 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:12.630218 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:12.633922 (+ 3704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:12.633933 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:12.634771 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:12.635484 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:12.635502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.636957 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.636961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:12.638333 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:12.638345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.639275 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.639283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:12.639495 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:12.643583 (+ 4088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:12.643607 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:12.646016 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:12.649280 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:12.649308 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:12.649323 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:12.649882 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:12.650294 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:12.650300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:12.650304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:12.650579 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:12.651377 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:12.651385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:12.651745 (+ 360us) client_negotiation.cc:769] Sending connection context
0928 11:02:12.652001 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 11:02:12.652524 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":114,"spinlock_wait_cycles":14208,"thread_start_us":303,"threads_started":1}
I20260928 11:02:12.653370 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:12.629148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:52349 (local address 127.3.252.67:37811)
0928 11:02:12.631461 (+ 2313us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:12.631479 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:12.631530 (+ 51us) server_negotiation.cc:407] Connection header received
0928 11:02:12.633037 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:12.633047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:12.633436 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:12.633732 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:12.635669 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.636788 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.638459 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.639056 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.639219 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:12.646339 (+ 7120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:12.646348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:12.646362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:12.646401 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:12.649135 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:12.650018 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:12.650025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:12.650031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:12.650188 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:12.650713 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:12.650717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:12.650719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:12.651241 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:12.651381 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:12.652585 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0928 11:02:12.652991 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2118,"thread_start_us":2015,"threads_started":1}
I20260928 11:02:12.653924 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:12.630199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:47991 (local address 127.3.252.66:43971)
0928 11:02:12.631826 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:12.631841 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:12.632176 (+ 335us) server_negotiation.cc:407] Connection header received
0928 11:02:12.633729 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:12.633736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:12.634088 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:12.634335 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:12.635666 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.636790 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.638456 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:12.639039 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:12.639195 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:12.646526 (+ 7331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:12.646533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:12.646546 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:12.646586 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:12.649224 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:12.650030 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:12.650036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:12.650041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:12.650182 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:12.650962 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:12.650965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:12.650968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:12.651508 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:12.651692 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:12.652248 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 11:02:12.653083 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":1309,"threads_started":1}
W20260928 11:02:12.656559 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
W20260928 11:02:12.656829 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:02:12.656919 20774 leader_election.cc:302] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb; no voters: cefbcea617ea4016a72abd5a8df9e859, e3c014f68f534cfb95da27367857b48b
I20260928 11:02:12.657249 21156 raft_consensus.cc:2726] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 11:02:12.769723 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:12.768904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:37759)
0928 11:02:12.769349 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:12.769489 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":211,"threads_started":1}
I20260928 11:02:12.910888 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:12.910478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:60415)
0928 11:02:12.910662 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:12.910753 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 11:02:13.341465 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:13.340784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:47435)
0928 11:02:13.341222 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:13.341312 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":232,"threads_started":1}
I20260928 11:02:13.771364 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:13.770651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:36035)
0928 11:02:13.771119 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:13.771205 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":248,"threads_started":1}
I20260928 11:02:13.912489 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:13.911767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:33297)
0928 11:02:13.912221 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:13.912308 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260928 11:02:14.319985 21165 raft_consensus.cc:482] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:14.320188 21165 raft_consensus.cc:504] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:02:14.320885 21165 leader_election.cc:288] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
W20260928 11:02:14.321933 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
W20260928 11:02:14.322162 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:02:14.322253 20774 leader_election.cc:302] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb; no voters: cefbcea617ea4016a72abd5a8df9e859, e3c014f68f534cfb95da27367857b48b
I20260928 11:02:14.322546 21165 raft_consensus.cc:2726] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 11:02:14.343140 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:14.342445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:47115)
0928 11:02:14.342883 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:14.342971 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260928 11:02:14.772984 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:14.772264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:50375)
0928 11:02:14.772743 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:14.772828 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":264,"threads_started":1}
I20260928 11:02:14.914206 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:14.913375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:33535)
0928 11:02:14.913946 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:14.914036 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":283,"threads_started":1}
I20260928 11:02:15.345139 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:15.344301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:33225)
0928 11:02:15.344849 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:15.344945 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":287,"threads_started":1}
I20260928 11:02:15.774822 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:15.773992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:54781)
0928 11:02:15.774468 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:15.774606 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":226,"threads_started":1}
I20260928 11:02:15.916044 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:15.915189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:60987)
0928 11:02:15.915740 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:15.915850 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":228,"threads_started":1}
I20260928 11:02:16.044548 21172 raft_consensus.cc:482] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:16.044742 21172 raft_consensus.cc:504] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:02:16.045356 21172 leader_election.cc:288] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
W20260928 11:02:16.046463 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
W20260928 11:02:16.046792 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:02:16.046929 20774 leader_election.cc:302] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb; no voters: cefbcea617ea4016a72abd5a8df9e859, e3c014f68f534cfb95da27367857b48b
I20260928 11:02:16.047189 21172 raft_consensus.cc:2726] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 11:02:16.347122 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:16.346360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:34195)
0928 11:02:16.346859 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:16.346949 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
I20260928 11:02:16.776650 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:16.775857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:54907)
0928 11:02:16.776355 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:16.776483 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
I20260928 11:02:16.917752 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:16.917036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:34173)
0928 11:02:16.917459 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:16.917549 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260928 11:02:17.348850 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:17.348046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:60579)
0928 11:02:17.348572 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:17.348667 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":270,"threads_started":1}
I20260928 11:02:17.778451 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:17.777645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:58663)
0928 11:02:17.778184 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:17.778274 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260928 11:02:17.919436 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:17.918716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:59959)
0928 11:02:17.919156 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:17.919242 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":218,"threads_started":1}
I20260928 11:02:17.946563 21179 raft_consensus.cc:482] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:17.946795 21179 raft_consensus.cc:504] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:02:17.947541 21179 leader_election.cc:288] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
W20260928 11:02:17.948488 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
W20260928 11:02:17.948720 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:02:17.948869 20774 leader_election.cc:302] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb; no voters: cefbcea617ea4016a72abd5a8df9e859, e3c014f68f534cfb95da27367857b48b
I20260928 11:02:17.949110 21179 raft_consensus.cc:2726] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 11:02:18.350762 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:18.349896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:36559)
0928 11:02:18.350450 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:18.350550 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":241,"threads_started":1}
I20260928 11:02:18.780290 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:18.779437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:32831)
0928 11:02:18.780004 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:18.780098 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260928 11:02:18.921368 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:18.920460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:34333)
0928 11:02:18.921062 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:18.921182 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":346,"threads_started":1}
I20260928 11:02:19.352433 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:19.351716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:51685)
0928 11:02:19.352164 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:19.352263 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260928 11:02:19.781993 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:19.781285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:36867)
0928 11:02:19.781701 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:19.781785 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260928 11:02:19.923149 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:19.922409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:48687)
0928 11:02:19.922871 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:19.922971 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260928 11:02:20.354480 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:20.353619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:38363)
0928 11:02:20.354208 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:20.354310 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":317,"threads_started":1}
I20260928 11:02:20.783612 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:20.782895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:50351)
0928 11:02:20.783358 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:20.783442 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260928 11:02:20.924826 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:20.924051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:49173)
0928 11:02:20.924533 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:20.924631 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
I20260928 11:02:21.356443 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:21.355748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:43747)
0928 11:02:21.356194 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:21.356276 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260928 11:02:21.785375 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:21.784601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:48483)
0928 11:02:21.785104 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:21.785191 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":282,"threads_started":1}
I20260928 11:02:21.926424 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:21.925686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:58323)
0928 11:02:21.926165 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:21.926253 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
I20260928 11:02:22.358158 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:22.357336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:36281)
0928 11:02:22.357884 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:22.357973 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
I20260928 11:02:22.787073 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:22.786285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:33885)
0928 11:02:22.786792 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:22.786884 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":243,"threads_started":1}
I20260928 11:02:22.927975 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:22.927312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:53061)
0928 11:02:22.927725 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:22.927820 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":205,"threads_started":1}
I20260928 11:02:23.359758 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:23.358980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:50707)
0928 11:02:23.359476 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:23.359576 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":262,"threads_started":1}
I20260928 11:02:23.788738 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:23.788011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:42151)
0928 11:02:23.788475 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:23.788561 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260928 11:02:23.890549 21197 raft_consensus.cc:482] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:23.890772 21197 raft_consensus.cc:504] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbcea617ea4016a72abd5a8df9e859" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 37811 } } peers { permanent_uuid: "e3c014f68f534cfb95da27367857b48b" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43971 } } peers { permanent_uuid: "5cdd07284bdb46d198bffda9961855cb" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 33585 } }
I20260928 11:02:23.891454 21197 leader_election.cc:288] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811), e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971)
W20260928 11:02:23.892271 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cefbcea617ea4016a72abd5a8df9e859 (127.3.252.67:37811): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
W20260928 11:02:23.892472 20774 leader_election.cc:341] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3c014f68f534cfb95da27367857b48b (127.3.252.66:43971): Not found: Tablet not found: 10faf6f64bec4268b3aafed2f9b39e2c
I20260928 11:02:23.892581 20774 leader_election.cc:302] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cdd07284bdb46d198bffda9961855cb; no voters: cefbcea617ea4016a72abd5a8df9e859, e3c014f68f534cfb95da27367857b48b
I20260928 11:02:23.892827 21197 raft_consensus.cc:2726] T 10faf6f64bec4268b3aafed2f9b39e2c P 5cdd07284bdb46d198bffda9961855cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 11:02:23.929721 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:23.928818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:50491)
0928 11:02:23.929357 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:23.929493 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":267,"threads_started":1}
I20260928 11:02:24.361471 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:24.360723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:51849)
0928 11:02:24.361212 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:24.361308 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260928 11:02:24.790408 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:24.789646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:53155)
0928 11:02:24.790140 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:24.790242 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
I20260928 11:02:24.931282 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:24.930546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:46281)
0928 11:02:24.931024 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:24.931104 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
I20260928 11:02:25.363246 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:25.362460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:33137)
0928 11:02:25.362953 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:25.363051 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":257,"threads_started":1}
I20260928 11:02:25.791946 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:25.791250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:59425)
0928 11:02:25.791654 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:25.791744 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":218,"threads_started":1}
I20260928 11:02:25.933048 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:25.932371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.65:52889)
0928 11:02:25.932789 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:25.932867 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260928 11:02:26.365257 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:26.364486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.66:49327)
0928 11:02:26.364989 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:26.365094 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":234,"threads_started":1}
I20260928 11:02:26.780606 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 20717
I20260928 11:02:26.793792 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:26.793047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:37019 (local address 127.3.252.67:51997)
0928 11:02:26.793530 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:26.793628 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260928 11:02:26.805213 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:26.781902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.3.252.65:33585)
0928 11:02:26.782462 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:26.782470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:26.782499 (+ 29us) server_negotiation.cc:407] Connection header received
0928 11:02:26.782568 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:26.782575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:26.782794 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:26.782926 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:26.784327 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:26.785363 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:26.786841 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:26.787429 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:26.787575 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:26.799793 (+ 12218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:26.799810 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:26.799827 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:26.799874 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:26.802672 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:26.803364 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:26.803371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:26.803373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:26.803510 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:26.803849 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:26.803853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:26.803854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:26.804208 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:26.804437 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:26.804694 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 11:02:26.804897 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":223,"threads_started":1}
I20260928 11:02:26.806573 20883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:26.914086 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 20717
I20260928 11:02:26.928769 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 20891
I20260928 11:02:26.941118 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:26.929493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.3.252.66:43971)
0928 11:02:26.929954 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:26.929958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:26.929975 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:02:26.930029 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:26.930033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:26.930091 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:26.930171 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:26.931442 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:26.932193 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:26.933386 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:26.933799 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:26.933959 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:26.936273 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:26.936278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:26.936281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:26.936326 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:26.938815 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:26.939492 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:26.939497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:26.939499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:26.939639 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:26.939986 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:26.939989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:26.939991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:26.940251 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:26.940447 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:26.940623 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 11:02:26.940825 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":227,"threads_started":1}
I20260928 11:02:26.941818 21014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:27.024513 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 20891
I20260928 11:02:27.041872 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21022
I20260928 11:02:27.055526 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:27.042716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.3.252.67:37811)
0928 11:02:27.043318 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:27.043323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:27.043341 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:02:27.043394 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:27.043398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:27.043464 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:27.043560 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:27.044973 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:27.045774 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:27.047096 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:27.047555 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:27.047663 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:27.050292 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:27.050298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:27.050301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:27.050336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:27.052976 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:27.053648 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:27.053653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:27.053655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:27.053810 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:27.054234 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:27.054237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:27.054239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:27.054502 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:27.054725 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:27.054970 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 11:02:27.055150 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":335,"threads_started":1}
I20260928 11:02:27.056248 21145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:27.139081 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21022
2026-09-28T11:02:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63717 ms)
[----------] 30 tests from SecurityITest (291073 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:27 dist-test-slave-djjw krb5kdc[21215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:27 dist-test-slave-djjw krb5kdc[21215](info): set up 2 sockets
Sep 28 11:02:27 dist-test-slave-djjw krb5kdc[21215](info): commencing operation
krb5kdc: starting...
W20260928 11:02:28.170082 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:28 dist-test-slave-djjw krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:02:28.409678 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33017
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:33017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:28.736578 21228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:28.736871 21228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:28.736989 21228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:28.744158 21228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:28.744266 21228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:28.744328 21228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:28.744405 21228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:28.744454 21228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:28.754236 21228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:33017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:28.755537 21228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:28.756803 21228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:28.768075 21233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:28.768095 21236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:28.768711 21234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:28.768819 21228 server_base.cc:1034] running on GCE node
I20260928 11:02:28.769585 21228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:28.771015 21228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:28.772213 21228 hybrid_clock.cc:648] HybridClock initialized: now 1790593348772159 us; error 47 us; skew 500 ppm
I20260928 11:02:28.772568 21228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:28 dist-test-slave-djjw krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:28.775765 21228 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:02:28.777318 21228 webserver.cc:460] Webserver started at http://127.3.252.126:35491/ using document root <none> and password file <none>
I20260928 11:02:28.778040 21228 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:28.778158 21228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:28.778451 21228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:28.781455 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "8d5a7a2d016b429fba43065d4db7da68"
format_stamp: "Formatted at 2026-09-28 11:02:28 on dist-test-slave-djjw"
server_key: "f3a1794ce12e7ccc8fc751e35ab7e877"
server_key_iv: "7e49375263de6a2df34e8cac46f6aa4c"
server_key_version: "encryptionkey@0"
I20260928 11:02:28.782234 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "8d5a7a2d016b429fba43065d4db7da68"
format_stamp: "Formatted at 2026-09-28 11:02:28 on dist-test-slave-djjw"
server_key: "f3a1794ce12e7ccc8fc751e35ab7e877"
server_key_iv: "7e49375263de6a2df34e8cac46f6aa4c"
server_key_version: "encryptionkey@0"
I20260928 11:02:28.787557 21228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 11:02:28.791115 21243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:28.792472 21228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:28.792634 21228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "8d5a7a2d016b429fba43065d4db7da68"
format_stamp: "Formatted at 2026-09-28 11:02:28 on dist-test-slave-djjw"
server_key: "f3a1794ce12e7ccc8fc751e35ab7e877"
server_key_iv: "7e49375263de6a2df34e8cac46f6aa4c"
server_key_version: "encryptionkey@0"
I20260928 11:02:28.792819 21228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:28.810482 21228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:28.814461 21228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:28.814721 21228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:28.831576 21228 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:33017
I20260928 11:02:28.831596 21294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:33017 every 8 connection(s)
I20260928 11:02:28.832890 21228 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:28.833240 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21228
I20260928 11:02:28.833436 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:28.834282 4081 external_mini_cluster.cc:1428] Setting key d98b5366cb0456e6a5ed7bc9709dc25d
I20260928 11:02:28.837791 21295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 11:02:28 dist-test-slave-djjw krb5kdc[21215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:28.850844 21295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68: Bootstrap starting.
I20260928 11:02:28.854207 21295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:28.855430 21295 log.cc:824] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:28.857806 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:28.836379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.3.252.126:33017)
0928 11:02:28.837310 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:28.837325 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:28.837632 (+ 307us) server_negotiation.cc:407] Connection header received
0928 11:02:28.839461 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:28.839482 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:28.839860 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:28.840198 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:28.841944 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:28.843309 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:28.845133 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:28.845807 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:28.845980 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:28.850805 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:28.850812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:28.850831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:28.850902 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:28.853744 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:28.854599 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:28.854605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:28.854612 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:28.854777 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:28.855166 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:28.855168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:28.855170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:28.855694 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:28.855974 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:28.856458 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 11:02:28.856880 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":425,"threads_started":1}
I20260928 11:02:28.858422 21295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68: No bootstrap required, opened a new log
I20260928 11:02:28.866145 21295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5a7a2d016b429fba43065d4db7da68" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33017 } }
I20260928 11:02:28.866530 21295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:28.866619 21295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5a7a2d016b429fba43065d4db7da68, State: Initialized, Role: FOLLOWER
I20260928 11:02:28.867198 21295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5a7a2d016b429fba43065d4db7da68" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33017 } }
I20260928 11:02:28.867413 21295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:28.867530 21295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:28.867674 21295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:28.870154 21295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5a7a2d016b429fba43065d4db7da68" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33017 } }
I20260928 11:02:28.870646 21295 leader_election.cc:302] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d5a7a2d016b429fba43065d4db7da68; no voters:
I20260928 11:02:28.871224 21295 leader_election.cc:288] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:28.871408 21300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:28.872020 21300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [term 1 LEADER]: Becoming Leader. State: Replica: 8d5a7a2d016b429fba43065d4db7da68, State: Running, Role: LEADER
I20260928 11:02:28.872596 21300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5a7a2d016b429fba43065d4db7da68" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33017 } }
I20260928 11:02:28.873375 21295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:28.875257 21301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d5a7a2d016b429fba43065d4db7da68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5a7a2d016b429fba43065d4db7da68" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33017 } } }
I20260928 11:02:28.875664 21301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:28.876137 21302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d5a7a2d016b429fba43065d4db7da68. Latest consensus state: current_term: 1 leader_uuid: "8d5a7a2d016b429fba43065d4db7da68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5a7a2d016b429fba43065d4db7da68" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33017 } } }
I20260928 11:02:28.876438 21302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:28.882604 21307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:28.888237 21307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:28.896972 21307 catalog_manager.cc:1347] Generated new cluster ID: d875e72ec53d4d048318b4b6f0d0e381
I20260928 11:02:28.897118 21307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:28.915061 21307 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:28.916465 21307 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:28.926746 21307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68: Generated new TSK 0
I20260928 11:02:28.927824 21307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 11:02:28.997793 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33017
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:29.326568 21321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:29.326912 21321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:29.327041 21321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:29.334697 21321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:29.334838 21321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:29.334987 21321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:02:29.345484 21321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:29.346874 21321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:29.348163 21321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:29.360986 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:29.361043 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:29.361001 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:29.361816 21321 server_base.cc:1034] running on GCE node
I20260928 11:02:29.362473 21321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:29.363408 21321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:29.364609 21321 hybrid_clock.cc:648] HybridClock initialized: now 1790593349364563 us; error 44 us; skew 500 ppm
I20260928 11:02:29.364914 21321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:29 dist-test-slave-djjw krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:29.368083 21321 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:02:29.369529 21321 webserver.cc:460] Webserver started at http://127.3.252.65:35239/ using document root <none> and password file <none>
I20260928 11:02:29.370221 21321 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:29.370352 21321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:29.370641 21321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:29.373433 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "8fc1bacb543e4244aa96674231d21c90"
format_stamp: "Formatted at 2026-09-28 11:02:29 on dist-test-slave-djjw"
server_key: "389b8b3a7a41af9d1efb6a227d91ece6"
server_key_iv: "1da4706e60bf9f8d2e4d45a8b0241147"
server_key_version: "encryptionkey@0"
I20260928 11:02:29.374195 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "8fc1bacb543e4244aa96674231d21c90"
format_stamp: "Formatted at 2026-09-28 11:02:29 on dist-test-slave-djjw"
server_key: "389b8b3a7a41af9d1efb6a227d91ece6"
server_key_iv: "1da4706e60bf9f8d2e4d45a8b0241147"
server_key_version: "encryptionkey@0"
I20260928 11:02:29.379279 21321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 11:02:29.382799 21336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:29.384202 21321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 11:02:29.384366 21321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "8fc1bacb543e4244aa96674231d21c90"
format_stamp: "Formatted at 2026-09-28 11:02:29 on dist-test-slave-djjw"
server_key: "389b8b3a7a41af9d1efb6a227d91ece6"
server_key_iv: "1da4706e60bf9f8d2e4d45a8b0241147"
server_key_version: "encryptionkey@0"
I20260928 11:02:29.384552 21321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:29.448868 21321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:29.453091 21321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:29.453352 21321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:29.454355 21321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:29.455675 21321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:29.455750 21321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:29.455873 21321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:29.455924 21321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:29.485663 21321 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:39301
I20260928 11:02:29.485710 21448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:39301 every 8 connection(s)
I20260928 11:02:29.487002 21321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:02:29.495647 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21321
I20260928 11:02:29.495883 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:02:29.496513 4081 external_mini_cluster.cc:1428] Setting key 12b1a110506b85b734d1400857bbc6cc
Sep 28 11:02:29 dist-test-slave-djjw krb5kdc[21215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:29.507264 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:29.489518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:55011 (local address 127.3.252.126:33017)
0928 11:02:29.490005 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:29.490011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:29.490981 (+ 970us) server_negotiation.cc:407] Connection header received
0928 11:02:29.492757 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:29.492761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:29.492839 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:29.492937 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:29.494451 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.495212 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.496929 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.497441 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.497559 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:29.501298 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:29.501320 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:29.501323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:29.501357 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:29.504032 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:29.504881 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:29.504885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:29.504887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:29.505030 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:29.505446 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:29.505450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:29.505452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:29.505721 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:29.505870 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:29.506648 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 11:02:29.506905 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":208,"threads_started":1}
I20260928 11:02:29.508028 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:29.490098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33017 (local address 127.3.252.65:55011)
0928 11:02:29.490788 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:29.490863 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:29.492489 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:29.493140 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:29.493149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:29.493656 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:29.494278 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:29.494297 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.495404 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.495408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:29.496783 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:29.496796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.497684 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.497691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:29.498029 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:29.498700 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:29.498729 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:29.501062 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:29.504189 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:29.504198 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:29.504218 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:29.504777 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:29.505119 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:29.505123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:29.505126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:29.505356 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:29.505860 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:29.505869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:29.506262 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 11:02:29.506502 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 11:02:29.506930 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":278,"threads_started":1}
I20260928 11:02:29.509764 21449 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33017
I20260928 11:02:29.510174 21449 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:29.511116 21449 heartbeater.cc:511] Master 127.3.252.126:33017 requested a full tablet report, sending...
I20260928 11:02:29.513468 21260 ts_manager.cc:194] Registered new tserver with Master: 8fc1bacb543e4244aa96674231d21c90 (127.3.252.65:39301)
I20260928 11:02:29.515892 21260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:55011
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 11:02:29.549569 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33017
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:29.870077 21455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:29.870354 21455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:29.870476 21455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:29.877715 21455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:29.877851 21455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:29.878006 21455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:02:29.888329 21455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:29.889642 21455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:29.890931 21455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:29.903560 21460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:29.903591 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:29.903682 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:29.904400 21455 server_base.cc:1034] running on GCE node
I20260928 11:02:29.905046 21455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:29.906064 21455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:29.907311 21455 hybrid_clock.cc:648] HybridClock initialized: now 1790593349907260 us; error 43 us; skew 500 ppm
I20260928 11:02:29.907641 21455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:29 dist-test-slave-djjw krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:29.910748 21455 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:02:29.912279 21455 webserver.cc:460] Webserver started at http://127.3.252.66:39239/ using document root <none> and password file <none>
I20260928 11:02:29.912949 21455 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:29.913067 21455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:29.913367 21455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:29.916385 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "6ae08a881ca3449ba7ec9c4702222779"
format_stamp: "Formatted at 2026-09-28 11:02:29 on dist-test-slave-djjw"
server_key: "527520439e2b02da15f5d267b03c68c6"
server_key_iv: "a97299871764cab6d2bd56590e039f63"
server_key_version: "encryptionkey@0"
I20260928 11:02:29.917171 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "6ae08a881ca3449ba7ec9c4702222779"
format_stamp: "Formatted at 2026-09-28 11:02:29 on dist-test-slave-djjw"
server_key: "527520439e2b02da15f5d267b03c68c6"
server_key_iv: "a97299871764cab6d2bd56590e039f63"
server_key_version: "encryptionkey@0"
I20260928 11:02:29.922513 21455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260928 11:02:29.925954 21470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:29.927397 21455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:29.927534 21455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "6ae08a881ca3449ba7ec9c4702222779"
format_stamp: "Formatted at 2026-09-28 11:02:29 on dist-test-slave-djjw"
server_key: "527520439e2b02da15f5d267b03c68c6"
server_key_iv: "a97299871764cab6d2bd56590e039f63"
server_key_version: "encryptionkey@0"
I20260928 11:02:29.927711 21455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:29.948562 21455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:29.952881 21455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:29.953140 21455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:29.954125 21455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:29.955415 21455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:29.955492 21455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:29.955576 21455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:29.955617 21455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:29.985111 21455 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:42615
I20260928 11:02:29.985183 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:42615 every 8 connection(s)
I20260928 11:02:29.986526 21455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:02:29.993084 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21455
I20260928 11:02:29.993240 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:02:29.993683 4081 external_mini_cluster.cc:1428] Setting key 785f0a69b40128f03fdff84d9a1642ec
Sep 28 11:02:29 dist-test-slave-djjw krb5kdc[21215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:30.006892 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:29.989386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:58117 (local address 127.3.252.126:33017)
0928 11:02:29.989569 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:29.989573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:29.990716 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 11:02:29.992558 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:29.992564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:29.992643 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:29.992728 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:29.994388 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.995137 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.996822 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.997317 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.997431 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.000890 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:30.000895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:30.000898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:30.000933 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:30.003722 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.004598 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.004603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.004605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.004744 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.005173 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.005176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.005178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.005459 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:30.005567 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.006424 (+ 857us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.006612 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 11:02:30.008190 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:29.989789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33017 (local address 127.3.252.66:58117)
0928 11:02:29.990558 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:29.990595 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:29.992286 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:29.992972 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:29.992989 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:29.993566 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:29.994187 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:29.994212 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.995290 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.995294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:29.996679 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:29.996693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:29.997541 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:29.997548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:29.997799 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:29.998482 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:29.998516 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:30.000646 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:30.003866 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:30.003876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:30.003895 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:30.004452 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:30.004840 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:30.004845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:30.004848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:30.005080 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:30.005571 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:30.005580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:30.006040 (+ 460us) client_negotiation.cc:769] Sending connection context
0928 11:02:30.006333 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 11:02:30.006896 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":275,"threads_started":1}
I20260928 11:02:30.009855 21583 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33017
I20260928 11:02:30.010210 21583 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:30.011018 21583 heartbeater.cc:511] Master 127.3.252.126:33017 requested a full tablet report, sending...
I20260928 11:02:30.012722 21260 ts_manager.cc:194] Registered new tserver with Master: 6ae08a881ca3449ba7ec9c4702222779 (127.3.252.66:42615)
I20260928 11:02:30.013816 21260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:58117
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 11:02:30.048444 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33017
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:30.379858 21588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:30.380175 21588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:30.380316 21588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:30.388347 21588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:30.388501 21588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:30.388661 21588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:02:30.399101 21588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:30.400465 21588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:30.401738 21588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:30.414942 21594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:30.414902 21593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:30.414974 21596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:30.415781 21588 server_base.cc:1034] running on GCE node
I20260928 11:02:30.416437 21588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:30.417452 21588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:30.418679 21588 hybrid_clock.cc:648] HybridClock initialized: now 1790593350418622 us; error 47 us; skew 500 ppm
I20260928 11:02:30.419070 21588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:30 dist-test-slave-djjw krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:30.422492 21588 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:02:30.424113 21588 webserver.cc:460] Webserver started at http://127.3.252.67:33377/ using document root <none> and password file <none>
I20260928 11:02:30.424840 21588 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:30.424965 21588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:30.425267 21588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:30.428313 21588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "99a2f4c8b90d4fb2ae3351942a75711f"
format_stamp: "Formatted at 2026-09-28 11:02:30 on dist-test-slave-djjw"
server_key: "6b6dac2c688ccdebc8cf2ef5226ced76"
server_key_iv: "2ffed0feb830bdf0885472358ae59ce5"
server_key_version: "encryptionkey@0"
I20260928 11:02:30.429113 21588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "99a2f4c8b90d4fb2ae3351942a75711f"
format_stamp: "Formatted at 2026-09-28 11:02:30 on dist-test-slave-djjw"
server_key: "6b6dac2c688ccdebc8cf2ef5226ced76"
server_key_iv: "2ffed0feb830bdf0885472358ae59ce5"
server_key_version: "encryptionkey@0"
I20260928 11:02:30.434918 21588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 11:02:30.438629 21603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:30.440277 21588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:30.440452 21588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "99a2f4c8b90d4fb2ae3351942a75711f"
format_stamp: "Formatted at 2026-09-28 11:02:30 on dist-test-slave-djjw"
server_key: "6b6dac2c688ccdebc8cf2ef5226ced76"
server_key_iv: "2ffed0feb830bdf0885472358ae59ce5"
server_key_version: "encryptionkey@0"
I20260928 11:02:30.440655 21588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:30.466558 21588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:30.470731 21588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:30.471007 21588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:30.471992 21588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:30.473467 21588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:30.473551 21588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:30.473671 21588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:30.473721 21588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:30.506129 21588 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:33039
I20260928 11:02:30.506176 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:33039 every 8 connection(s)
I20260928 11:02:30.507329 21588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:02:30.512063 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21588
I20260928 11:02:30.512234 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:02:30.512756 4081 external_mini_cluster.cc:1428] Setting key 4147860642a6e7c1e2e504df0846c75c
I20260928 11:02:30.519127 21449 heartbeater.cc:503] Master 127.3.252.126:33017 was elected leader, sending a full tablet report...
Sep 28 11:02:30 dist-test-slave-djjw krb5kdc[21215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:30.527458 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.509726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:49213 (local address 127.3.252.126:33017)
0928 11:02:30.510213 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.510218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.511063 (+ 845us) server_negotiation.cc:407] Connection header received
0928 11:02:30.513450 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.513456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.513558 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.513665 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:30.515560 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.516435 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.517977 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.518360 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.518457 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.521914 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:30.521919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:30.521922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:30.521954 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:30.524337 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.525134 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.525138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.525140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.525265 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.525636 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.525639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.525641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.525950 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:30.526071 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.526832 (+ 761us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.527095 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":276,"threads_started":1}
I20260928 11:02:30.528723 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.510277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33017 (local address 127.3.252.67:49213)
0928 11:02:30.510911 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:30.510953 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:30.513153 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:30.514052 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:30.514060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:30.514727 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:30.515395 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.515419 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.516586 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.516590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.517824 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.517863 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.518549 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.518554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.518747 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.519422 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:30.519451 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:30.521687 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:30.524500 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:30.524508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:30.524521 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:30.524992 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:30.525364 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:30.525367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:30.525370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:30.525551 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:30.526056 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:30.526063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:30.526399 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 11:02:30.526672 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 11:02:30.527154 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":232,"threads_started":1}
I20260928 11:02:30.530627 21716 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33017
I20260928 11:02:30.530951 21716 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:30.531697 21716 heartbeater.cc:511] Master 127.3.252.126:33017 requested a full tablet report, sending...
I20260928 11:02:30.533138 21260 ts_manager.cc:194] Registered new tserver with Master: 99a2f4c8b90d4fb2ae3351942a75711f (127.3.252.67:33039)
I20260928 11:02:30.533993 21260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:49213
I20260928 11:02:30.541239 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:02:30.558282 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.545712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.3.252.126:33017)
0928 11:02:30.545961 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.545967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.546290 (+ 323us) server_negotiation.cc:407] Connection header received
0928 11:02:30.546460 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.546466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.546530 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.546609 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:30.548001 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.548705 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.550124 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.550532 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.550642 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.553132 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:30.553136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:30.553140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:30.553174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:30.555632 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.556338 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.556343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.556345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.556474 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.556816 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.556820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.556822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.557144 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:30.557374 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.557716 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.557974 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 11:02:30.562386 21260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:02:30.569446 21260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:02:30.591543 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.582012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.3.252.65:39301)
0928 11:02:30.582679 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.582687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.582717 (+ 30us) server_negotiation.cc:407] Connection header received
0928 11:02:30.582943 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.582950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.583200 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.583388 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:30.584101 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.585889 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.589198 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.590395 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.590521 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.590754 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.591045 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.591208 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":361,"threads_started":1}
I20260928 11:02:30.591598 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.581685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39301 (local address 127.0.0.1:48892)
0928 11:02:30.582546 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:30.582585 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:30.582720 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:30.583426 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:30.583434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:30.583467 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:30.583890 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.583905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.586031 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.586037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.589014 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.589027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.590603 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.590609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.590827 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.590869 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 11:02:30.590978 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 11:02:30.591180 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":2,"thread_start_us":980,"threads_started":1}
I20260928 11:02:30.593032 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.584091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:42615 (local address 127.0.0.1:40452)
0928 11:02:30.584561 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:30.584576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:30.584676 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:30.585884 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:30.585888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:30.585923 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:30.586369 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.586382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.588211 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.588215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.590503 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.590518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.592256 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.592262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.592417 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.592449 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 11:02:30.592542 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 11:02:30.592665 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":22,"thread_start_us":676,"threads_started":1}
I20260928 11:02:30.593050 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.584543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.3.252.66:42615)
0928 11:02:30.585231 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.585239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.585266 (+ 27us) server_negotiation.cc:407] Connection header received
0928 11:02:30.585368 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.585376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.585633 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.586232 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:30.586536 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.588051 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.590689 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.592051 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.592180 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.592348 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.592594 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.592741 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":1183,"threads_started":1}
I20260928 11:02:30.594380 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.583447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:33039 (local address 127.0.0.1:46270)
0928 11:02:30.584038 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:30.584054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:30.584152 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:30.586887 (+ 2735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:30.586892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:30.586914 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:30.587260 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.587269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.589086 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.589092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.591222 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.591235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.593516 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.593522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.593689 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.593721 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 11:02:30.593790 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 11:02:30.593934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":4,"thread_start_us":327,"threads_started":1}
I20260928 11:02:30.595508 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.583785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.3.252.67:33039)
0928 11:02:30.586449 (+ 2664us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.586454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.586477 (+ 23us) server_negotiation.cc:407] Connection header received
0928 11:02:30.586544 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.586549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.586698 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.586869 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:30.587425 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.588927 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.591406 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.593018 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.594742 (+ 1724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.594956 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.595019 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.595171 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2523,"thread_start_us":2328,"threads_started":1}
I20260928 11:02:30.595515 21384 tablet_service.cc:1467] Processing CreateTablet for tablet dbfa1ec4fadd4e2685039d10d28e8d06 (DEFAULT_TABLE table=test-table [id=d39528db847b443fabfd126e13e1e3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:30.596487 21518 tablet_service.cc:1467] Processing CreateTablet for tablet dbfa1ec4fadd4e2685039d10d28e8d06 (DEFAULT_TABLE table=test-table [id=d39528db847b443fabfd126e13e1e3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:30.596975 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfa1ec4fadd4e2685039d10d28e8d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:30.597595 21518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfa1ec4fadd4e2685039d10d28e8d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:30.599206 21651 tablet_service.cc:1467] Processing CreateTablet for tablet dbfa1ec4fadd4e2685039d10d28e8d06 (DEFAULT_TABLE table=test-table [id=d39528db847b443fabfd126e13e1e3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:30.600533 21651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfa1ec4fadd4e2685039d10d28e8d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:30.608629 21735 tablet_bootstrap.cc:492] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Bootstrap starting.
I20260928 11:02:30.611332 21736 tablet_bootstrap.cc:492] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Bootstrap starting.
I20260928 11:02:30.612345 21735 tablet_bootstrap.cc:654] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:30.613724 21735 log.cc:824] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:30.614902 21736 tablet_bootstrap.cc:654] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:30.616225 21736 log.cc:824] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:30.617084 21737 tablet_bootstrap.cc:492] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Bootstrap starting.
I20260928 11:02:30.619892 21736 tablet_bootstrap.cc:492] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: No bootstrap required, opened a new log
I20260928 11:02:30.619892 21735 tablet_bootstrap.cc:492] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: No bootstrap required, opened a new log
I20260928 11:02:30.620172 21736 ts_tablet_manager.cc:1397] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 11:02:30.620173 21735 ts_tablet_manager.cc:1397] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 11:02:30.620548 21737 tablet_bootstrap.cc:654] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:30.621462 21737 log.cc:824] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:30.624212 21737 tablet_bootstrap.cc:492] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: No bootstrap required, opened a new log
I20260928 11:02:30.624452 21737 ts_tablet_manager.cc:1397] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260928 11:02:30.630574 21735 raft_consensus.cc:348] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.630574 21736 raft_consensus.cc:348] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.631035 21736 raft_consensus.cc:374] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:30.631035 21735 raft_consensus.cc:374] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:30.631137 21735 raft_consensus.cc:729] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae08a881ca3449ba7ec9c4702222779, State: Initialized, Role: FOLLOWER
I20260928 11:02:30.631138 21736 raft_consensus.cc:729] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc1bacb543e4244aa96674231d21c90, State: Initialized, Role: FOLLOWER
I20260928 11:02:30.631280 21737 raft_consensus.cc:348] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.631742 21737 raft_consensus.cc:374] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:30.631834 21737 raft_consensus.cc:729] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a2f4c8b90d4fb2ae3351942a75711f, State: Initialized, Role: FOLLOWER
I20260928 11:02:30.631774 21736 consensus_queue.cc:260] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.631783 21735 consensus_queue.cc:260] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.632462 21737 consensus_queue.cc:260] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.633563 21735 ts_tablet_manager.cc:1428] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 11:02:30.633563 21736 ts_tablet_manager.cc:1428] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 11:02:30.634718 21583 heartbeater.cc:503] Master 127.3.252.126:33017 was elected leader, sending a full tablet report...
I20260928 11:02:30.638602 21716 heartbeater.cc:503] Master 127.3.252.126:33017 was elected leader, sending a full tablet report...
I20260928 11:02:30.639351 21737 ts_tablet_manager.cc:1428] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Time spent starting tablet: real 0.015s user 0.004s sys 0.011s
I20260928 11:02:30.649358 21743 raft_consensus.cc:482] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:30.649607 21743 raft_consensus.cc:504] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.651183 21743 leader_election.cc:288] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fc1bacb543e4244aa96674231d21c90 (127.3.252.65:39301), 6ae08a881ca3449ba7ec9c4702222779 (127.3.252.66:42615)
I20260928 11:02:30.656296 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.651577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:52889 (local address 127.3.252.65:39301)
0928 11:02:30.651751 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.651767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.651786 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:02:30.651859 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.651864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.651947 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.652037 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:30.652452 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.653177 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.654735 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.655472 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.655609 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.655682 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.655924 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.656046 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 11:02:30.656296 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.651377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39301 (local address 127.3.252.67:52889)
0928 11:02:30.651556 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:30.651575 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:30.651683 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:30.652106 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:30.652111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:30.652149 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:30.652315 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.652325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.653303 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.653307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.654587 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.654602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.655605 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.655610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.655778 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.655802 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 11:02:30.655881 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 11:02:30.656007 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 11:02:30.657075 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.651966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:47149 (local address 127.3.252.66:42615)
0928 11:02:30.652155 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.652161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.652443 (+ 282us) server_negotiation.cc:407] Connection header received
0928 11:02:30.652587 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.652591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.652644 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.652714 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:30.653172 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.653866 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.655461 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.656216 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.656355 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.656421 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.656718 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.656845 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 11:02:30.657102 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.651723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:42615 (local address 127.3.252.67:47149)
0928 11:02:30.652354 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:30.652368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:30.652473 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:30.652735 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:30.652738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:30.652761 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:30.653045 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.653053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.653993 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.653998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.655316 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:30.655327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.656399 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.656405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:30.656571 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.656603 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 11:02:30.656675 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 11:02:30.656790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":443,"threads_started":1}
I20260928 11:02:30.657282 21404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa1ec4fadd4e2685039d10d28e8d06" candidate_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc1bacb543e4244aa96674231d21c90" is_pre_election: true
I20260928 11:02:30.657817 21404 raft_consensus.cc:2454] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a2f4c8b90d4fb2ae3351942a75711f in term 0.
I20260928 11:02:30.657943 21538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa1ec4fadd4e2685039d10d28e8d06" candidate_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae08a881ca3449ba7ec9c4702222779" is_pre_election: true
I20260928 11:02:30.658449 21538 raft_consensus.cc:2454] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a2f4c8b90d4fb2ae3351942a75711f in term 0.
I20260928 11:02:30.658473 21607 leader_election.cc:302] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fc1bacb543e4244aa96674231d21c90, 99a2f4c8b90d4fb2ae3351942a75711f; no voters:
I20260928 11:02:30.658828 21743 raft_consensus.cc:2781] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:02:30.658974 21743 raft_consensus.cc:482] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:02:30.659077 21743 raft_consensus.cc:3037] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:30.661211 21743 raft_consensus.cc:504] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.661736 21743 leader_election.cc:288] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [CANDIDATE]: Term 1 election: Requested vote from peers 8fc1bacb543e4244aa96674231d21c90 (127.3.252.65:39301), 6ae08a881ca3449ba7ec9c4702222779 (127.3.252.66:42615)
I20260928 11:02:30.662185 21404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa1ec4fadd4e2685039d10d28e8d06" candidate_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc1bacb543e4244aa96674231d21c90"
I20260928 11:02:30.662250 21538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa1ec4fadd4e2685039d10d28e8d06" candidate_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae08a881ca3449ba7ec9c4702222779"
I20260928 11:02:30.662357 21404 raft_consensus.cc:3037] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:30.662464 21538 raft_consensus.cc:3037] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:30.664709 21404 raft_consensus.cc:2454] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a2f4c8b90d4fb2ae3351942a75711f in term 1.
I20260928 11:02:30.664891 21538 raft_consensus.cc:2454] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a2f4c8b90d4fb2ae3351942a75711f in term 1.
I20260928 11:02:30.665089 21607 leader_election.cc:302] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fc1bacb543e4244aa96674231d21c90, 99a2f4c8b90d4fb2ae3351942a75711f; no voters:
I20260928 11:02:30.665349 21743 raft_consensus.cc:2781] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:30.665743 21743 raft_consensus.cc:686] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 1 LEADER]: Becoming Leader. State: Replica: 99a2f4c8b90d4fb2ae3351942a75711f, State: Running, Role: LEADER
I20260928 11:02:30.666280 21743 consensus_queue.cc:237] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } }
I20260928 11:02:30.670583 21258 catalog_manager.cc:5495] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a2f4c8b90d4fb2ae3351942a75711f (127.3.252.67). New cstate: current_term: 1 leader_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a2f4c8b90d4fb2ae3351942a75711f" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 } health_report { overall_health: UNKNOWN } } }
I20260928 11:02:30.696393 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.691206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.3.252.67:33039)
0928 11:02:30.691390 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.691394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.691411 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:02:30.691493 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.691496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.691580 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.691699 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:02:30.692181 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.692939 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.694373 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.694776 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.694876 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.695341 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:02:30.695862 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:02:30.695971 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.696101 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.696172 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 11:02:30.704473 21538 raft_consensus.cc:1261] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 1 FOLLOWER]: Refusing update from remote peer 99a2f4c8b90d4fb2ae3351942a75711f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:30.704502 21404 raft_consensus.cc:1261] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 1 FOLLOWER]: Refusing update from remote peer 99a2f4c8b90d4fb2ae3351942a75711f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:30.705426 21745 consensus_queue.cc:1035] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae08a881ca3449ba7ec9c4702222779" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 42615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:30.705770 21743 consensus_queue.cc:1035] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc1bacb543e4244aa96674231d21c90" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:30.730429 21258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:30.730695 21258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:30.734387 21258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d5a7a2d016b429fba43065d4db7da68: Sending DeleteTablet for 3 replicas of tablet dbfa1ec4fadd4e2685039d10d28e8d06
I20260928 11:02:30.735525 21384 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfa1ec4fadd4e2685039d10d28e8d06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:30 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:48892
I20260928 11:02:30.735651 21518 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfa1ec4fadd4e2685039d10d28e8d06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:30 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:40452
I20260928 11:02:30.736642 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfa1ec4fadd4e2685039d10d28e8d06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:30 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:46270
I20260928 11:02:30.736869 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21321
I20260928 11:02:30.746024 21759 tablet_replica.cc:330] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: stopping tablet replica
I20260928 11:02:30.746628 21759 raft_consensus.cc:2229] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:30.747279 21759 raft_consensus.cc:2258] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:30.748279 21760 tablet_replica.cc:330] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: stopping tablet replica
I20260928 11:02:30.748695 21760 raft_consensus.cc:2229] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:02:30.749353 21760 raft_consensus.cc:2258] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:30.751075 21759 ts_tablet_manager.cc:1905] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:30.760275 21760 ts_tablet_manager.cc:1905] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:30.763316 21758 tablet_replica.cc:330] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: stopping tablet replica
I20260928 11:02:30.763706 21758 raft_consensus.cc:2229] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:30.764175 21758 raft_consensus.cc:2258] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:30.766211 21758 ts_tablet_manager.cc:1905] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:30.767284 21759 ts_tablet_manager.cc:1918] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:30.767467 21759 log.cc:1196] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/dbfa1ec4fadd4e2685039d10d28e8d06
I20260928 11:02:30.768006 21759 ts_tablet_manager.cc:1939] T dbfa1ec4fadd4e2685039d10d28e8d06 P 6ae08a881ca3449ba7ec9c4702222779: Deleting consensus metadata
I20260928 11:02:30.768450 21760 ts_tablet_manager.cc:1918] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:30.768616 21760 log.cc:1196] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/dbfa1ec4fadd4e2685039d10d28e8d06
I20260928 11:02:30.769153 21760 ts_tablet_manager.cc:1939] T dbfa1ec4fadd4e2685039d10d28e8d06 P 99a2f4c8b90d4fb2ae3351942a75711f: Deleting consensus metadata
I20260928 11:02:30.770027 21247 catalog_manager.cc:4841] TS 6ae08a881ca3449ba7ec9c4702222779 (127.3.252.66:42615): tablet dbfa1ec4fadd4e2685039d10d28e8d06 (table test-table [id=d39528db847b443fabfd126e13e1e3d8]) successfully deleted
Sep 28 11:02:30 dist-test-slave-djjw krb5kdc[21215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 11:02:30.771617 21247 catalog_manager.cc:4841] TS 99a2f4c8b90d4fb2ae3351942a75711f (127.3.252.67:33039): tablet dbfa1ec4fadd4e2685039d10d28e8d06 (table test-table [id=d39528db847b443fabfd126e13e1e3d8]) successfully deleted
I20260928 11:02:30.773217 21758 ts_tablet_manager.cc:1918] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:30.773382 21758 log.cc:1196] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/dbfa1ec4fadd4e2685039d10d28e8d06
I20260928 11:02:30.773779 21758 ts_tablet_manager.cc:1939] T dbfa1ec4fadd4e2685039d10d28e8d06 P 8fc1bacb543e4244aa96674231d21c90: Deleting consensus metadata
I20260928 11:02:30.775307 21247 catalog_manager.cc:4841] TS 8fc1bacb543e4244aa96674231d21c90 (127.3.252.65:39301): tablet dbfa1ec4fadd4e2685039d10d28e8d06 (table test-table [id=d39528db847b443fabfd126e13e1e3d8]) successfully deleted
I20260928 11:02:30.779541 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.738463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.3.252.65:39301)
0928 11:02:30.738655 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.738662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.754486 (+ 15824us) server_negotiation.cc:407] Connection header received
0928 11:02:30.754785 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.754792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.754964 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.755047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:30.757061 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.758468 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.761052 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.761813 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.761954 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.772421 (+ 10467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:30.772437 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:30.772457 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:30.772507 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:30.776488 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.777296 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.777306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.777310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.777489 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.777891 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.777897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.777900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.778362 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:30.778650 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.778924 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.779156 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:02:30.780197 21444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:30.869436 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21321
I20260928 11:02:30.884326 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21455
Sep 28 11:02:30 dist-test-slave-djjw krb5kdc[21215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 11:02:30.901235 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:30.885122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.3.252.66:42615)
0928 11:02:30.885297 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:30.885301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:30.885336 (+ 35us) server_negotiation.cc:407] Connection header received
0928 11:02:30.885490 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:30.885494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:30.885559 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:30.885640 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:30.887278 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.887998 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.889402 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:30.889789 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:30.890093 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:30.894868 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:30.894885 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:30.894902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:30.894947 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:30.898179 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.899101 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.899109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.899112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.899248 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:30.899685 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:30.899691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:30.899694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:30.900112 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:30.900344 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:30.900759 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 11:02:30.900943 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 11:02:30.901926 21578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:30.989804 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21455
I20260928 11:02:31.004354 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21588
Sep 28 11:02:31 dist-test-slave-djjw krb5kdc[21215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 11:02:31.021967 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:31.005217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.3.252.67:33039)
0928 11:02:31.005393 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:31.005398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:31.005415 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:02:31.005493 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:31.005496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:31.005557 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:31.005644 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:31.007456 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:31.008280 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:31.010196 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:31.010778 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:31.010896 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:31.016142 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:31.016157 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:31.016173 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:31.016219 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:31.019138 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:31.019826 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:31.019835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:31.019837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:31.020000 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:31.020364 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:31.020368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:31.020370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:31.020749 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:31.021040 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:31.021375 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 11:02:31.021597 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 11:02:31.022621 21711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:31.144290 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21588
I20260928 11:02:31.160234 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21228
I20260928 11:02:31.161304 21290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:31.270903 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21228
2026-09-28T11:02:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4133 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T11:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:31Z Disabled control of system clock
I20260928 11:02:31.341349 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40589
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:40589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 11:02:31.665503 21769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:31.665812 21769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:31.665946 21769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:31.673416 21769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:31.673528 21769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:31.673566 21769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:31.673641 21769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:31.673693 21769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:31.683204 21769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:40589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:31.684525 21769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:31.686339 21769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:31.697899 21774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:31.697875 21777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:31.697808 21775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:31.698222 21769 server_base.cc:1034] running on GCE node
I20260928 11:02:31.699221 21769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:31.700567 21769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:31.701776 21769 hybrid_clock.cc:648] HybridClock initialized: now 1790593351701727 us; error 34 us; skew 500 ppm
I20260928 11:02:31.702158 21769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:31.704453 21769 webserver.cc:460] Webserver started at http://127.3.252.126:41569/ using document root <none> and password file <none>
I20260928 11:02:31.705127 21769 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:31.705219 21769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:31.705535 21769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:31.708422 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "9cbc24670e554dd5838a449ef202dda9"
format_stamp: "Formatted at 2026-09-28 11:02:31 on dist-test-slave-djjw"
server_key: "bb0a303b48e4805ccc424d668afe66f7"
server_key_iv: "508be27196a9bed752fd267177f47581"
server_key_version: "encryptionkey@0"
I20260928 11:02:31.709182 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "9cbc24670e554dd5838a449ef202dda9"
format_stamp: "Formatted at 2026-09-28 11:02:31 on dist-test-slave-djjw"
server_key: "bb0a303b48e4805ccc424d668afe66f7"
server_key_iv: "508be27196a9bed752fd267177f47581"
server_key_version: "encryptionkey@0"
I20260928 11:02:31.714362 21769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 11:02:31.717952 21783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:31.719463 21769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:31.719650 21769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "9cbc24670e554dd5838a449ef202dda9"
format_stamp: "Formatted at 2026-09-28 11:02:31 on dist-test-slave-djjw"
server_key: "bb0a303b48e4805ccc424d668afe66f7"
server_key_iv: "508be27196a9bed752fd267177f47581"
server_key_version: "encryptionkey@0"
I20260928 11:02:31.719877 21769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:31.743436 21769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:31.744303 21769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:31.744588 21769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:31.762435 21769 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:40589
I20260928 11:02:31.762455 21834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:40589 every 8 connection(s)
I20260928 11:02:31.763768 21769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:31.764411 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21769
I20260928 11:02:31.764588 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:31.765127 4081 external_mini_cluster.cc:1428] Setting key 91201a1162ceaa76e668674ca0d44cdd
I20260928 11:02:31.768332 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:31.781036 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:31.767006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.3.252.126:40589)
0928 11:02:31.768013 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:31.768028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:31.768079 (+ 51us) server_negotiation.cc:407] Connection header received
0928 11:02:31.770346 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:31.770375 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:31.770931 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:31.771397 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:31.771943 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:31.773972 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:31.775426 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:31.776478 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:31.776671 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:31.777913 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:31.777925 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:31.777949 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:31.778716 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:31.778814 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:31.778851 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:31.778992 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:31.779304 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 11:02:31.779683 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":307,"threads_started":1}
I20260928 11:02:31.782197 21835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9: Bootstrap starting.
I20260928 11:02:31.785305 21835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:31.786370 21835 log.cc:824] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:31.789556 21835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9: No bootstrap required, opened a new log
I20260928 11:02:31.797302 21835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbc24670e554dd5838a449ef202dda9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40589 } }
I20260928 11:02:31.797691 21835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:31.797787 21835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbc24670e554dd5838a449ef202dda9, State: Initialized, Role: FOLLOWER
I20260928 11:02:31.798382 21835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbc24670e554dd5838a449ef202dda9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40589 } }
I20260928 11:02:31.798594 21835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:31.798724 21835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:31.798882 21835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:31.801554 21835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbc24670e554dd5838a449ef202dda9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40589 } }
I20260928 11:02:31.802114 21835 leader_election.cc:302] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cbc24670e554dd5838a449ef202dda9; no voters:
I20260928 11:02:31.802671 21835 leader_election.cc:288] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:31.802815 21840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:31.803654 21840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [term 1 LEADER]: Becoming Leader. State: Replica: 9cbc24670e554dd5838a449ef202dda9, State: Running, Role: LEADER
I20260928 11:02:31.804253 21840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbc24670e554dd5838a449ef202dda9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40589 } }
I20260928 11:02:31.805052 21835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:31.807034 21841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cbc24670e554dd5838a449ef202dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbc24670e554dd5838a449ef202dda9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40589 } } }
I20260928 11:02:31.807479 21841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:31.807384 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cbc24670e554dd5838a449ef202dda9. Latest consensus state: current_term: 1 leader_uuid: "9cbc24670e554dd5838a449ef202dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbc24670e554dd5838a449ef202dda9" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40589 } } }
I20260928 11:02:31.807660 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:31.809664 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:31.814975 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:31.824354 21848 catalog_manager.cc:1347] Generated new cluster ID: 44f382a16bd04118b7b02c27f13dd886
I20260928 11:02:31.824478 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:31.839712 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:31.841699 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:31.856204 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9: Generated new TSK 0
I20260928 11:02:31.857362 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:31.871333 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40589
--builtin_ntp_servers=127.3.252.84:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 11:02:32.206839 21859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:32.207150 21859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:32.207265 21859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:32.214838 21859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:32.214948 21859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:32.215080 21859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:02:32.224517 21859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:32.225791 21859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:32.227676 21859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:32.241315 21864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:32.241339 21867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:32.241415 21865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:32.242434 21859 server_base.cc:1034] running on GCE node
I20260928 11:02:32.243052 21859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:32.244139 21859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:32.245407 21859 hybrid_clock.cc:648] HybridClock initialized: now 1790593352245346 us; error 49 us; skew 500 ppm
I20260928 11:02:32.245724 21859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:32.248271 21859 webserver.cc:460] Webserver started at http://127.3.252.65:37457/ using document root <none> and password file <none>
I20260928 11:02:32.248950 21859 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:32.249047 21859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:32.249325 21859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:32.252257 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "51c6ed760a4d4dc1877499353d28e749"
format_stamp: "Formatted at 2026-09-28 11:02:32 on dist-test-slave-djjw"
server_key: "cf6b3405fee642676887ab1b163afda0"
server_key_iv: "a5c5c09a34ebe590b05bb1b4f18f29df"
server_key_version: "encryptionkey@0"
I20260928 11:02:32.253031 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "51c6ed760a4d4dc1877499353d28e749"
format_stamp: "Formatted at 2026-09-28 11:02:32 on dist-test-slave-djjw"
server_key: "cf6b3405fee642676887ab1b163afda0"
server_key_iv: "a5c5c09a34ebe590b05bb1b4f18f29df"
server_key_version: "encryptionkey@0"
I20260928 11:02:32.258143 21859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 11:02:32.261685 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:32.263230 21859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:32.263374 21859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "51c6ed760a4d4dc1877499353d28e749"
format_stamp: "Formatted at 2026-09-28 11:02:32 on dist-test-slave-djjw"
server_key: "cf6b3405fee642676887ab1b163afda0"
server_key_iv: "a5c5c09a34ebe590b05bb1b4f18f29df"
server_key_version: "encryptionkey@0"
I20260928 11:02:32.263582 21859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:32.288076 21859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:32.288892 21859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:32.289170 21859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:32.290139 21859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:32.291738 21859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:32.291826 21859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:32.291904 21859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:32.291960 21859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:32.323611 21859 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:36303
I20260928 11:02:32.323698 21985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:36303 every 8 connection(s)
I20260928 11:02:32.324934 21859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:02:32.325635 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21859
I20260928 11:02:32.325811 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:02:32.326448 4081 external_mini_cluster.cc:1428] Setting key e5411e2fd4cc684d42ad81313c10d78a
I20260928 11:02:32.330204 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40589
--builtin_ntp_servers=127.3.252.84:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 11:02:32.347913 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:32.328312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:36721 (local address 127.3.252.126:40589)
0928 11:02:32.334299 (+ 5987us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:32.334308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:32.338883 (+ 4575us) server_negotiation.cc:407] Connection header received
0928 11:02:32.340743 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:32.340748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:32.340845 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:32.340956 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:32.342167 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.343207 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.344815 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.345371 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.345490 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:32.346899 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:32.346906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:32.346910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:32.346933 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:32.346981 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:32.346996 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:32.347079 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:32.347361 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 11:02:32.347521 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5818,"thread_start_us":5686,"threads_started":1}
I20260928 11:02:32.348791 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:32.329198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40589 (local address 127.3.252.65:36721)
0928 11:02:32.338688 (+ 9490us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:32.338743 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:32.340452 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:32.341212 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:32.341221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:32.341325 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:32.341986 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:32.342006 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.343383 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.343387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:32.344677 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:32.344689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.345602 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.345607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:32.345808 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:32.346547 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:32.346549 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:32.346592 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:32.346680 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:32.347126 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:32.347136 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:32.347143 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 11:02:32.347309 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 11:02:32.347650 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8967,"thread_start_us":8740,"threads_started":1}
I20260928 11:02:32.350718 21986 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40589
I20260928 11:02:32.351080 21986 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:32.352066 21986 heartbeater.cc:511] Master 127.3.252.126:40589 requested a full tablet report, sending...
I20260928 11:02:32.354651 21800 ts_manager.cc:194] Registered new tserver with Master: 51c6ed760a4d4dc1877499353d28e749 (127.3.252.65:36303)
I20260928 11:02:32.356292 21800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.65:36721
W20260928 11:02:32.661707 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:32.662055 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:32.662189 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:32.669505 21990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:32.669615 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:32.669754 21990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:02:32.678918 21990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:32.680197 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:32.681988 21990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:32.694707 21998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:32.694689 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:32.694975 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:32.695675 21990 server_base.cc:1034] running on GCE node
I20260928 11:02:32.696270 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:32.697245 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:32.698451 21990 hybrid_clock.cc:648] HybridClock initialized: now 1790593352698399 us; error 48 us; skew 500 ppm
I20260928 11:02:32.698733 21990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:32.700949 21990 webserver.cc:460] Webserver started at http://127.3.252.66:41335/ using document root <none> and password file <none>
I20260928 11:02:32.701596 21990 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:32.701685 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:32.702024 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:32.704792 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "33df44b101d340958e800fb16036f6d7"
format_stamp: "Formatted at 2026-09-28 11:02:32 on dist-test-slave-djjw"
server_key: "edec78718369c54e823faa02309c18d2"
server_key_iv: "e80a7edce8c17ad8f3fe2cce223c61dd"
server_key_version: "encryptionkey@0"
I20260928 11:02:32.705561 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "33df44b101d340958e800fb16036f6d7"
format_stamp: "Formatted at 2026-09-28 11:02:32 on dist-test-slave-djjw"
server_key: "edec78718369c54e823faa02309c18d2"
server_key_iv: "e80a7edce8c17ad8f3fe2cce223c61dd"
server_key_version: "encryptionkey@0"
I20260928 11:02:32.710695 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 11:02:32.714221 22004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:32.715626 21990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:02:32.715781 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "33df44b101d340958e800fb16036f6d7"
format_stamp: "Formatted at 2026-09-28 11:02:32 on dist-test-slave-djjw"
server_key: "edec78718369c54e823faa02309c18d2"
server_key_iv: "e80a7edce8c17ad8f3fe2cce223c61dd"
server_key_version: "encryptionkey@0"
I20260928 11:02:32.715984 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:32.738564 21990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:32.739367 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:32.739650 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:32.740602 21990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:32.742061 21990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:32.742153 21990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:32.742255 21990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:32.742321 21990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:32.773638 21990 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:43937
I20260928 11:02:32.773700 22116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:43937 every 8 connection(s)
I20260928 11:02:32.774919 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:02:32.778693 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 21990
I20260928 11:02:32.778863 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:02:32.779356 4081 external_mini_cluster.cc:1428] Setting key c7c6525ba943ef64a81580281ab632f8
I20260928 11:02:32.782058 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40589
--builtin_ntp_servers=127.3.252.84:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 11:02:32.787734 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:32.777482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:37301 (local address 127.3.252.126:40589)
0928 11:02:32.777691 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:32.777698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:32.779178 (+ 1480us) server_negotiation.cc:407] Connection header received
0928 11:02:32.781395 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:32.781400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:32.781477 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:32.781559 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:32.782705 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.783400 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.784935 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.785318 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.785433 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:32.786928 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:32.786933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:32.786936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:32.786956 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:32.786988 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:32.787000 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:32.787074 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:32.787295 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 11:02:32.787424 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 11:02:32.788697 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:32.777981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40589 (local address 127.3.252.66:37301)
0928 11:02:32.779026 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:32.779069 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:32.781106 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:32.781791 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:32.781799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:32.781931 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:32.782548 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:32.782566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.783545 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.783551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:32.784770 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:32.784782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:32.785538 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:32.785546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:32.785773 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:32.786590 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:32.786593 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:32.786632 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:32.786719 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:32.787086 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:32.787096 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:32.787105 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 11:02:32.787254 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 11:02:32.787661 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":324,"threads_started":1}
I20260928 11:02:32.790130 22117 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40589
I20260928 11:02:32.790437 22117 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:32.791182 22117 heartbeater.cc:511] Master 127.3.252.126:40589 requested a full tablet report, sending...
I20260928 11:02:32.792654 21800 ts_manager.cc:194] Registered new tserver with Master: 33df44b101d340958e800fb16036f6d7 (127.3.252.66:43937)
I20260928 11:02:32.793496 21800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.66:37301
W20260928 11:02:33.098074 22120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:33.098373 22120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:33.098490 22120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:33.105760 22120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:33.105906 22120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:33.106055 22120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:02:33.115789 22120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:40589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:33.117098 22120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:33.118909 22120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:33.131857 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:33.131829 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:33.131894 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:33.132689 22120 server_base.cc:1034] running on GCE node
I20260928 11:02:33.133327 22120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:33.134368 22120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:33.135586 22120 hybrid_clock.cc:648] HybridClock initialized: now 1790593353135531 us; error 48 us; skew 500 ppm
I20260928 11:02:33.135929 22120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:33.138242 22120 webserver.cc:460] Webserver started at http://127.3.252.67:41133/ using document root <none> and password file <none>
I20260928 11:02:33.138936 22120 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:33.139037 22120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:33.139358 22120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:33.142261 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "dd38cddd553749ccb24509df62b929d1"
format_stamp: "Formatted at 2026-09-28 11:02:33 on dist-test-slave-djjw"
server_key: "803740b55dbd971e644e016407e82169"
server_key_iv: "68d8176a9ecac01b988f1a912c6826f9"
server_key_version: "encryptionkey@0"
I20260928 11:02:33.143054 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "dd38cddd553749ccb24509df62b929d1"
format_stamp: "Formatted at 2026-09-28 11:02:33 on dist-test-slave-djjw"
server_key: "803740b55dbd971e644e016407e82169"
server_key_iv: "68d8176a9ecac01b988f1a912c6826f9"
server_key_version: "encryptionkey@0"
I20260928 11:02:33.148128 22120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 11:02:33.151684 22134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:33.153139 22120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:02:33.153306 22120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "dd38cddd553749ccb24509df62b929d1"
format_stamp: "Formatted at 2026-09-28 11:02:33 on dist-test-slave-djjw"
server_key: "803740b55dbd971e644e016407e82169"
server_key_iv: "68d8176a9ecac01b988f1a912c6826f9"
server_key_version: "encryptionkey@0"
I20260928 11:02:33.153554 22120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:33.169966 22120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:33.170789 22120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:33.171123 22120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:33.172149 22120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:33.173606 22120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:33.173704 22120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:33.173808 22120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:33.173906 22120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 11:02:33.204478 22120 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:40919
I20260928 11:02:33.204524 22246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:40919 every 8 connection(s)
I20260928 11:02:33.205760 22120 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:02:33.215627 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 22120
I20260928 11:02:33.215794 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:02:33.216207 4081 external_mini_cluster.cc:1428] Setting key aa1d6a9f7797bd344e642b4e2dc20b43
I20260928 11:02:33.218140 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.208387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:38291 (local address 127.3.252.126:40589)
0928 11:02:33.208568 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.208572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.209920 (+ 1348us) server_negotiation.cc:407] Connection header received
0928 11:02:33.211777 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.211782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.211863 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.211946 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:33.213001 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.213879 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.215401 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.215839 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.215958 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.217256 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:33.217263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:33.217268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:33.217289 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:33.217341 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:33.217357 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:33.217447 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.217709 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.217807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 11:02:33.219275 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.208864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:40589 (local address 127.3.252.67:38291)
0928 11:02:33.209760 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:33.209796 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:33.211510 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:33.212190 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:33.212199 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:33.212285 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:33.212832 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.212849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.214024 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.214028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.215261 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.215274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.216055 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.216062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.216272 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.216932 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:33.216934 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:33.216974 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:33.217077 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:33.217468 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:33.217479 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:33.217487 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 11:02:33.217684 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 11:02:33.218107 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":269,"threads_started":1}
I20260928 11:02:33.220747 22247 heartbeater.cc:347] Connected to a master server at 127.3.252.126:40589
I20260928 11:02:33.221035 22247 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:33.221858 22247 heartbeater.cc:511] Master 127.3.252.126:40589 requested a full tablet report, sending...
I20260928 11:02:33.223393 21800 ts_manager.cc:194] Registered new tserver with Master: dd38cddd553749ccb24509df62b929d1 (127.3.252.67:40919)
I20260928 11:02:33.224192 21800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.67:38291
I20260928 11:02:33.231673 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:02:33.241628 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.235631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.3.252.126:40589)
0928 11:02:33.235894 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.235901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.236227 (+ 326us) server_negotiation.cc:407] Connection header received
0928 11:02:33.236362 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.236366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.236447 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.236536 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:33.237064 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.238034 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.239464 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.239879 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.239990 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.240958 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:33.240970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:33.240977 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:33.240996 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:33.241029 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:33.241041 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:33.241123 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.241238 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.241318 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 11:02:33.245561 21800 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:02:33.252638 21800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:02:33.277634 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.264829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:36303 (local address 127.0.0.1:43500)
0928 11:02:33.266714 (+ 1885us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:33.266733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:33.266871 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:33.267924 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:33.267931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:33.267961 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:33.268387 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.268400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.270426 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.270432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.272852 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.272875 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.275472 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.275478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.276847 (+ 1369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.276926 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 11:02:33.277078 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 11:02:33.277267 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1761,"thread_start_us":2278,"threads_started":1}
I20260928 11:02:33.278254 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.265232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.3.252.65:36303)
0928 11:02:33.266279 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.266289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.267006 (+ 717us) server_negotiation.cc:407] Connection header received
0928 11:02:33.267122 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.267132 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.267437 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.267631 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:33.268598 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.270232 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.273466 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.274954 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.275082 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.275346 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.277691 (+ 2345us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.277927 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":746,"threads_started":1}
I20260928 11:02:33.281070 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.264608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:43937 (local address 127.0.0.1:59804)
0928 11:02:33.265458 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:33.265506 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:33.265669 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:33.274140 (+ 8471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:33.274145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:33.274175 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:33.274573 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.274585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.276599 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.276604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.278638 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.278652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.280332 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.280339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.280512 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.280544 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 11:02:33.280634 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 11:02:33.280758 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":490,"threads_started":1}
I20260928 11:02:33.281181 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.264983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.3.252.66:43937)
0928 11:02:33.273452 (+ 8469us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.273461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.273490 (+ 29us) server_negotiation.cc:407] Connection header received
0928 11:02:33.273590 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.273598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.273902 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.274779 (+ 877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:33.275028 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.276458 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.278831 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.280139 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.280273 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.280531 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.280684 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.280834 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8291,"thread_start_us":8181,"threads_started":1}
I20260928 11:02:33.282217 21921 tablet_service.cc:1467] Processing CreateTablet for tablet 8023363a7917429599d9923d501ed861 (DEFAULT_TABLE table=test-table [id=c00303fdc3724502a7ed060213a18619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:33.283502 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8023363a7917429599d9923d501ed861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:33.284667 22052 tablet_service.cc:1467] Processing CreateTablet for tablet 8023363a7917429599d9923d501ed861 (DEFAULT_TABLE table=test-table [id=c00303fdc3724502a7ed060213a18619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:33.286119 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8023363a7917429599d9923d501ed861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:33.286540 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.266410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40919 (local address 127.0.0.1:38108)
0928 11:02:33.277687 (+ 11277us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:33.277705 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:33.278205 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:33.278998 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:33.279003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:33.279026 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:33.279313 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.279322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.281605 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.281609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.283489 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.283506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.285676 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.285683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.285891 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.285926 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 11:02:33.286013 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 11:02:33.286159 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11136,"thread_start_us":10991,"threads_started":1}
I20260928 11:02:33.286631 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.266755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.3.252.67:40919)
0928 11:02:33.277219 (+ 10464us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.277229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.278092 (+ 863us) server_negotiation.cc:407] Connection header received
0928 11:02:33.278543 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.278553 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.278820 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.278995 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:33.279486 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.281455 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.283751 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.285419 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.285646 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.286034 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.286115 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.286280 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10054,"thread_start_us":10194,"threads_started":1}
I20260928 11:02:33.290704 22182 tablet_service.cc:1467] Processing CreateTablet for tablet 8023363a7917429599d9923d501ed861 (DEFAULT_TABLE table=test-table [id=c00303fdc3724502a7ed060213a18619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:33.292230 22182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8023363a7917429599d9923d501ed861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:33.297618 22265 tablet_bootstrap.cc:492] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Bootstrap starting.
I20260928 11:02:33.300902 22265 tablet_bootstrap.cc:654] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:33.302217 22265 log.cc:824] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:33.305739 22265 tablet_bootstrap.cc:492] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: No bootstrap required, opened a new log
I20260928 11:02:33.306089 22265 ts_tablet_manager.cc:1397] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 11:02:33.306126 22267 tablet_bootstrap.cc:492] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Bootstrap starting.
I20260928 11:02:33.307775 22268 tablet_bootstrap.cc:492] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Bootstrap starting.
I20260928 11:02:33.310216 22267 tablet_bootstrap.cc:654] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:33.311115 22268 tablet_bootstrap.cc:654] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:33.311540 22267 log.cc:824] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:33.312058 22268 log.cc:824] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:33.314942 22267 tablet_bootstrap.cc:492] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: No bootstrap required, opened a new log
I20260928 11:02:33.315161 22268 tablet_bootstrap.cc:492] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: No bootstrap required, opened a new log
I20260928 11:02:33.315263 22267 ts_tablet_manager.cc:1397] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 11:02:33.315511 22268 ts_tablet_manager.cc:1397] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 11:02:33.315869 22265 raft_consensus.cc:348] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.316311 22265 raft_consensus.cc:374] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:33.316407 22265 raft_consensus.cc:729] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c6ed760a4d4dc1877499353d28e749, State: Initialized, Role: FOLLOWER
I20260928 11:02:33.316988 22265 consensus_queue.cc:260] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.318114 21986 heartbeater.cc:503] Master 127.3.252.126:40589 was elected leader, sending a full tablet report...
I20260928 11:02:33.318809 22265 ts_tablet_manager.cc:1428] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260928 11:02:33.326220 22268 raft_consensus.cc:348] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.326203 22267 raft_consensus.cc:348] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.326669 22268 raft_consensus.cc:374] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:33.326669 22267 raft_consensus.cc:374] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:33.326776 22268 raft_consensus.cc:729] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd38cddd553749ccb24509df62b929d1, State: Initialized, Role: FOLLOWER
I20260928 11:02:33.326776 22267 raft_consensus.cc:729] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33df44b101d340958e800fb16036f6d7, State: Initialized, Role: FOLLOWER
I20260928 11:02:33.327456 22268 consensus_queue.cc:260] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.327456 22267 consensus_queue.cc:260] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.328631 22247 heartbeater.cc:503] Master 127.3.252.126:40589 was elected leader, sending a full tablet report...
I20260928 11:02:33.329139 22117 heartbeater.cc:503] Master 127.3.252.126:40589 was elected leader, sending a full tablet report...
I20260928 11:02:33.329777 22267 ts_tablet_manager.cc:1428] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 11:02:33.330632 22268 ts_tablet_manager.cc:1428] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260928 11:02:33.333444 21987 tablet.cc:2369] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:02:33.366072 22273 raft_consensus.cc:482] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:33.366394 22273 raft_consensus.cc:504] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.368263 22273 leader_election.cc:288] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51c6ed760a4d4dc1877499353d28e749 (127.3.252.65:36303), dd38cddd553749ccb24509df62b929d1 (127.3.252.67:40919)
I20260928 11:02:33.375003 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.368754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:51163 (local address 127.3.252.67:40919)
0928 11:02:33.368968 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.368974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.369553 (+ 579us) server_negotiation.cc:407] Connection header received
0928 11:02:33.369739 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.369743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.369805 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.369912 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:33.370506 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.371222 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.373209 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.374055 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.374200 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.374273 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.374629 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.374756 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 11:02:33.375012 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.368626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:40333 (local address 127.3.252.65:36303)
0928 11:02:33.368820 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.368826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.369615 (+ 789us) server_negotiation.cc:407] Connection header received
0928 11:02:33.369768 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.369772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.369874 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.369963 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:33.370550 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.371203 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.373233 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.374065 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.374229 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.374294 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.374631 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.374762 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 11:02:33.375308 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.368615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:36303 (local address 127.3.252.66:40333)
0928 11:02:33.369512 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:33.369526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:33.369644 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:33.369985 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:33.369990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:33.370018 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:33.370398 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.370407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.371332 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.371337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.373014 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.373027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.374264 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.374270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.374481 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.374505 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 11:02:33.374583 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 11:02:33.374753 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":113,"thread_start_us":524,"threads_started":1}
I20260928 11:02:33.375377 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.368533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40919 (local address 127.3.252.66:51163)
0928 11:02:33.369440 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:33.369456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:33.369598 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:33.369930 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:33.369935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:33.369960 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:33.370363 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.370374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.371345 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.371349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.373014 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:33.373027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.374256 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.374262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:33.374476 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.374502 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 11:02:33.374576 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 11:02:33.374753 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":105,"thread_start_us":525,"threads_started":1}
I20260928 11:02:33.376329 21941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023363a7917429599d9923d501ed861" candidate_uuid: "33df44b101d340958e800fb16036f6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c6ed760a4d4dc1877499353d28e749" is_pre_election: true
I20260928 11:02:33.376479 22202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023363a7917429599d9923d501ed861" candidate_uuid: "33df44b101d340958e800fb16036f6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd38cddd553749ccb24509df62b929d1" is_pre_election: true
I20260928 11:02:33.376878 21941 raft_consensus.cc:2454] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33df44b101d340958e800fb16036f6d7 in term 0.
I20260928 11:02:33.376976 22202 raft_consensus.cc:2454] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33df44b101d340958e800fb16036f6d7 in term 0.
I20260928 11:02:33.377645 22007 leader_election.cc:302] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33df44b101d340958e800fb16036f6d7, dd38cddd553749ccb24509df62b929d1; no voters:
I20260928 11:02:33.378104 22273 raft_consensus.cc:2781] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:02:33.378228 22273 raft_consensus.cc:482] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:02:33.378319 22273 raft_consensus.cc:3037] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:33.380475 22273 raft_consensus.cc:504] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.381085 22273 leader_election.cc:288] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [CANDIDATE]: Term 1 election: Requested vote from peers 51c6ed760a4d4dc1877499353d28e749 (127.3.252.65:36303), dd38cddd553749ccb24509df62b929d1 (127.3.252.67:40919)
I20260928 11:02:33.381511 21941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023363a7917429599d9923d501ed861" candidate_uuid: "33df44b101d340958e800fb16036f6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c6ed760a4d4dc1877499353d28e749"
I20260928 11:02:33.381610 22202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023363a7917429599d9923d501ed861" candidate_uuid: "33df44b101d340958e800fb16036f6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd38cddd553749ccb24509df62b929d1"
I20260928 11:02:33.381728 21941 raft_consensus.cc:3037] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:33.381798 22202 raft_consensus.cc:3037] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:33.384912 22202 raft_consensus.cc:2454] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33df44b101d340958e800fb16036f6d7 in term 1.
I20260928 11:02:33.384912 21941 raft_consensus.cc:2454] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33df44b101d340958e800fb16036f6d7 in term 1.
I20260928 11:02:33.385376 22005 leader_election.cc:302] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33df44b101d340958e800fb16036f6d7, 51c6ed760a4d4dc1877499353d28e749; no voters:
I20260928 11:02:33.385708 22273 raft_consensus.cc:2781] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:33.386202 22273 raft_consensus.cc:686] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 1 LEADER]: Becoming Leader. State: Replica: 33df44b101d340958e800fb16036f6d7, State: Running, Role: LEADER
I20260928 11:02:33.386680 22273 consensus_queue.cc:237] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } }
I20260928 11:02:33.391503 21798 catalog_manager.cc:5495] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33df44b101d340958e800fb16036f6d7 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "33df44b101d340958e800fb16036f6d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df44b101d340958e800fb16036f6d7" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 } health_report { overall_health: UNKNOWN } } }
I20260928 11:02:33.433115 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.427928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.3.252.66:43937)
0928 11:02:33.428122 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.428127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.428144 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:02:33.428237 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.428240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.428324 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.428460 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:02:33.428797 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.429545 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.430984 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.431411 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.431510 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.432024 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:02:33.432533 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:02:33.432669 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.432760 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.432827 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 11:02:33.441531 22202 raft_consensus.cc:1261] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 1 FOLLOWER]: Refusing update from remote peer 33df44b101d340958e800fb16036f6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:33.441530 21941 raft_consensus.cc:1261] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 1 FOLLOWER]: Refusing update from remote peer 33df44b101d340958e800fb16036f6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:33.442788 22276 consensus_queue.cc:1035] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c6ed760a4d4dc1877499353d28e749" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 36303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:33.443207 22273 consensus_queue.cc:1035] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd38cddd553749ccb24509df62b929d1" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:33.457942 22281 mvcc.cc:205] Tried to move back new op lower bound from 7334270375685550080 to 7334270375475744768. Current Snapshot: MvccSnapshot[applied={T|T < 7334270375685550080}]
I20260928 11:02:33.470438 21800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:33.470702 21800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:33.474350 21800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cbc24670e554dd5838a449ef202dda9: Sending DeleteTablet for 3 replicas of tablet 8023363a7917429599d9923d501ed861
I20260928 11:02:33.475479 22052 tablet_service.cc:1514] Processing DeleteTablet for tablet 8023363a7917429599d9923d501ed861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:33 UTC) from {username='slave'} at 127.0.0.1:59804
I20260928 11:02:33.475584 21921 tablet_service.cc:1514] Processing DeleteTablet for tablet 8023363a7917429599d9923d501ed861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:33 UTC) from {username='slave'} at 127.0.0.1:43500
I20260928 11:02:33.476204 22182 tablet_service.cc:1514] Processing DeleteTablet for tablet 8023363a7917429599d9923d501ed861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:33 UTC) from {username='slave'} at 127.0.0.1:38108
I20260928 11:02:33.482105 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21859
I20260928 11:02:33.487701 22289 tablet_replica.cc:330] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: stopping tablet replica
I20260928 11:02:33.488109 22289 raft_consensus.cc:2229] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:33.488562 22289 raft_consensus.cc:2258] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:33.489249 22290 tablet_replica.cc:330] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: stopping tablet replica
I20260928 11:02:33.489571 22290 raft_consensus.cc:2229] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:02:33.490141 22290 raft_consensus.cc:2258] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:33.492082 22289 ts_tablet_manager.cc:1905] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:33.492274 22290 ts_tablet_manager.cc:1905] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:33.499497 22289 ts_tablet_manager.cc:1918] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:33.499784 22289 log.cc:1196] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/8023363a7917429599d9923d501ed861
I20260928 11:02:33.500350 22289 ts_tablet_manager.cc:1939] T 8023363a7917429599d9923d501ed861 P 51c6ed760a4d4dc1877499353d28e749: Deleting consensus metadata
I20260928 11:02:33.502573 21784 catalog_manager.cc:4841] TS 51c6ed760a4d4dc1877499353d28e749 (127.3.252.65:36303): tablet 8023363a7917429599d9923d501ed861 (table test-table [id=c00303fdc3724502a7ed060213a18619]) successfully deleted
I20260928 11:02:33.505100 22290 ts_tablet_manager.cc:1918] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:33.505190 22291 tablet_replica.cc:330] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: stopping tablet replica
I20260928 11:02:33.505296 22290 log.cc:1196] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/8023363a7917429599d9923d501ed861
I20260928 11:02:33.505666 22291 raft_consensus.cc:2229] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:33.505806 22290 ts_tablet_manager.cc:1939] T 8023363a7917429599d9923d501ed861 P 33df44b101d340958e800fb16036f6d7: Deleting consensus metadata
I20260928 11:02:33.506945 22291 raft_consensus.cc:2258] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:33.507475 21787 catalog_manager.cc:4841] TS 33df44b101d340958e800fb16036f6d7 (127.3.252.66:43937): tablet 8023363a7917429599d9923d501ed861 (table test-table [id=c00303fdc3724502a7ed060213a18619]) successfully deleted
I20260928 11:02:33.509135 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.483501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.3.252.65:36303)
0928 11:02:33.483685 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.483690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.500471 (+ 16781us) server_negotiation.cc:407] Connection header received
0928 11:02:33.500748 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.500754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.500870 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.500952 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:33.501596 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.502742 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.504638 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.505222 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.505344 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.507692 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:33.507704 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:33.507721 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:33.508290 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:33.508344 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:33.508364 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:33.508533 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.508649 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.508757 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 11:02:33.509986 22291 ts_tablet_manager.cc:1905] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:33.510025 21981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:33.517467 22291 ts_tablet_manager.cc:1918] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:33.517717 22291 log.cc:1196] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/8023363a7917429599d9923d501ed861
I20260928 11:02:33.518306 22291 ts_tablet_manager.cc:1939] T 8023363a7917429599d9923d501ed861 P dd38cddd553749ccb24509df62b929d1: Deleting consensus metadata
I20260928 11:02:33.520426 21786 catalog_manager.cc:4841] TS dd38cddd553749ccb24509df62b929d1 (127.3.252.67:40919): tablet 8023363a7917429599d9923d501ed861 (table test-table [id=c00303fdc3724502a7ed060213a18619]) successfully deleted
I20260928 11:02:33.602715 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21859
I20260928 11:02:33.624841 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21990
I20260928 11:02:33.632373 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.625604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.3.252.66:43937)
0928 11:02:33.625772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.625776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.625795 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:02:33.626024 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.626028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.626105 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.626199 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:33.626648 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.627382 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.629231 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.629650 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.629743 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.631048 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:33.631057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:33.631070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:33.631658 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:33.631709 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:33.631729 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:33.631804 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.631961 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.632041 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 11:02:33.633127 22112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:33.742894 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21990
I20260928 11:02:33.758653 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 22120
I20260928 11:02:33.766036 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:33.759477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.3.252.67:40919)
0928 11:02:33.759627 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:33.759632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:33.759649 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:02:33.759706 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:33.759710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:33.759787 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:33.759918 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:33.760319 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.761114 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.762792 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:33.763257 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:33.763365 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:33.764490 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:33.764499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:33.764524 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:33.765120 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:33.765174 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:33.765196 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:33.765287 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:33.765463 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 11:02:33.765563 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 11:02:33.766683 22242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:33.853847 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 22120
I20260928 11:02:33.868075 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 21769
I20260928 11:02:33.869127 21830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:33.977150 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 21769
2026-09-28T11:02:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2706 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:34 dist-test-slave-djjw krb5kdc[22299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:34 dist-test-slave-djjw krb5kdc[22299](info): set up 2 sockets
Sep 28 11:02:34 dist-test-slave-djjw krb5kdc[22299](info): commencing operation
krb5kdc: starting...
W20260928 11:02:34.998091 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:35 dist-test-slave-djjw krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:02:35.241487 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36185
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:35.589502 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:35.589805 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:35.589952 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:35.597214 22312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:35.597329 22312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:35.597384 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:35.597465 22312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:35.597520 22312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:35.607599 22312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:35.608916 22312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:35.610224 22312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:35.621570 22317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:35.621569 22320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:35.621531 22318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:35.622479 22312 server_base.cc:1034] running on GCE node
I20260928 11:02:35.623235 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:35.624598 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:35.625805 22312 hybrid_clock.cc:648] HybridClock initialized: now 1790593355625748 us; error 44 us; skew 500 ppm
I20260928 11:02:35.626173 22312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:35 dist-test-slave-djjw krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:35.629104 22312 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:02:35.630612 22312 webserver.cc:460] Webserver started at http://127.3.252.126:44909/ using document root <none> and password file <none>
I20260928 11:02:35.631316 22312 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:35.631441 22312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:35.631734 22312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:35.634763 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "a7c4b06d4a2d4e878492a111ab121dc8"
format_stamp: "Formatted at 2026-09-28 11:02:35 on dist-test-slave-djjw"
server_key: "bf176ec4a1490fda438a70f83d79bf29"
server_key_iv: "79c6c1640992b2b51ee93a47e16ac85e"
server_key_version: "encryptionkey@0"
I20260928 11:02:35.635525 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "a7c4b06d4a2d4e878492a111ab121dc8"
format_stamp: "Formatted at 2026-09-28 11:02:35 on dist-test-slave-djjw"
server_key: "bf176ec4a1490fda438a70f83d79bf29"
server_key_iv: "79c6c1640992b2b51ee93a47e16ac85e"
server_key_version: "encryptionkey@0"
I20260928 11:02:35.640692 22312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 11:02:35.644296 22327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:35.645660 22312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 11:02:35.645852 22312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "a7c4b06d4a2d4e878492a111ab121dc8"
format_stamp: "Formatted at 2026-09-28 11:02:35 on dist-test-slave-djjw"
server_key: "bf176ec4a1490fda438a70f83d79bf29"
server_key_iv: "79c6c1640992b2b51ee93a47e16ac85e"
server_key_version: "encryptionkey@0"
I20260928 11:02:35.646071 22312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:35.667924 22312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:35.672254 22312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:35.672530 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:35.689713 22312 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:36185
I20260928 11:02:35.689733 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:36185 every 8 connection(s)
I20260928 11:02:35.691088 22312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:35.694924 22379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:35.696344 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 22312
I20260928 11:02:35.696542 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:35.697100 4081 external_mini_cluster.cc:1428] Setting key 953d44ee8b6325f069a05ad217539503
I20260928 11:02:35.707998 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8: Bootstrap starting.
Sep 28 11:02:35 dist-test-slave-djjw krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:35.711290 22379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:35.712492 22379 log.cc:824] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:35.715875 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8: No bootstrap required, opened a new log
I20260928 11:02:35.719619 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:35.699107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.3.252.126:36185)
0928 11:02:35.699897 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:35.699910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:35.699951 (+ 41us) server_negotiation.cc:407] Connection header received
0928 11:02:35.701607 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:35.701635 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:35.702097 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:35.702483 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:35.703934 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:35.705337 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:35.706702 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:35.707372 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:35.707554 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:35.711765 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:35.711779 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:35.711796 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:35.711835 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:35.715020 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:35.715914 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:35.715924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:35.715933 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:35.716140 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:35.716595 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:35.716599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:35.716601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:35.717200 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:35.717447 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:35.718057 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 11:02:35.718508 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":358,"threads_started":1}
I20260928 11:02:35.724241 22379 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36185 } }
I20260928 11:02:35.724638 22379 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:35.724736 22379 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c4b06d4a2d4e878492a111ab121dc8, State: Initialized, Role: FOLLOWER
I20260928 11:02:35.725339 22379 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36185 } }
I20260928 11:02:35.725580 22379 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:35.725708 22379 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:35.725895 22379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:35.728128 22379 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36185 } }
I20260928 11:02:35.728680 22379 leader_election.cc:302] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7c4b06d4a2d4e878492a111ab121dc8; no voters:
I20260928 11:02:35.729314 22379 leader_election.cc:288] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:35.729465 22384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:35.730108 22384 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [term 1 LEADER]: Becoming Leader. State: Replica: a7c4b06d4a2d4e878492a111ab121dc8, State: Running, Role: LEADER
I20260928 11:02:35.730649 22384 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36185 } }
I20260928 11:02:35.731318 22379 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:35.733548 22386 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c4b06d4a2d4e878492a111ab121dc8. Latest consensus state: current_term: 1 leader_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36185 } } }
I20260928 11:02:35.733990 22386 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:35.736044 22391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:35.735900 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c4b06d4a2d4e878492a111ab121dc8" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36185 } } }
I20260928 11:02:35.736259 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:35.741147 22391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:35.750602 22391 catalog_manager.cc:1347] Generated new cluster ID: 639f55bf2c554d088b8d13d3b3769203
I20260928 11:02:35.750728 22391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:35.766080 22391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:35.767417 22391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:35.774386 22391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8: Generated new TSK 0
I20260928 11:02:35.775234 22391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 11:02:35.835182 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36185
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:36.162858 22405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:36.163138 22405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:36.163260 22405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:36.170471 22405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:36.170581 22405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:36.170727 22405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:02:36.180531 22405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:36.181804 22405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:36.183120 22405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:36.195454 22410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:36.195411 22411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:36.195526 22413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:36.196252 22405 server_base.cc:1034] running on GCE node
I20260928 11:02:36.196815 22405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:36.197701 22405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:36.198909 22405 hybrid_clock.cc:648] HybridClock initialized: now 1790593356198842 us; error 63 us; skew 500 ppm
I20260928 11:02:36.199267 22405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:36 dist-test-slave-djjw krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:36.202320 22405 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:02:36.203691 22405 webserver.cc:460] Webserver started at http://127.3.252.65:44155/ using document root <none> and password file <none>
I20260928 11:02:36.204320 22405 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:36.204413 22405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:36.204670 22405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:36.207489 22405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "23ea5616bf154253a8d876f22d58c460"
format_stamp: "Formatted at 2026-09-28 11:02:36 on dist-test-slave-djjw"
server_key: "9bf066e771874feb4f25e1a8ff3734d7"
server_key_iv: "4b842cffea9fc9777decae75c00ddf14"
server_key_version: "encryptionkey@0"
I20260928 11:02:36.208199 22405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "23ea5616bf154253a8d876f22d58c460"
format_stamp: "Formatted at 2026-09-28 11:02:36 on dist-test-slave-djjw"
server_key: "9bf066e771874feb4f25e1a8ff3734d7"
server_key_iv: "4b842cffea9fc9777decae75c00ddf14"
server_key_version: "encryptionkey@0"
I20260928 11:02:36.213160 22405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 11:02:36.216717 22420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:36.218248 22405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:02:36.218401 22405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "23ea5616bf154253a8d876f22d58c460"
format_stamp: "Formatted at 2026-09-28 11:02:36 on dist-test-slave-djjw"
server_key: "9bf066e771874feb4f25e1a8ff3734d7"
server_key_iv: "4b842cffea9fc9777decae75c00ddf14"
server_key_version: "encryptionkey@0"
I20260928 11:02:36.218586 22405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:36.233996 22405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:36.238468 22405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:36.238747 22405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:36.239728 22405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:36.241122 22405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:36.241194 22405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:36.241307 22405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:36.241361 22405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:36.271682 22405 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:41507
I20260928 11:02:36.271721 22532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:41507 every 8 connection(s)
I20260928 11:02:36.272960 22405 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:02:36.281746 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 22405
I20260928 11:02:36.281963 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:02:36.282575 4081 external_mini_cluster.cc:1428] Setting key b1da4ccd5bad65c1650fcb82d51d1efd
Sep 28 11:02:36 dist-test-slave-djjw krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:36.294910 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:36.275606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:57477 (local address 127.3.252.126:36185)
0928 11:02:36.276060 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:36.276067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:36.277296 (+ 1229us) server_negotiation.cc:407] Connection header received
0928 11:02:36.279199 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:36.279204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:36.279283 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:36.279382 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:36.280915 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.281953 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.283709 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.284318 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.284450 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:36.288570 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:36.288578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:36.288581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:36.288617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:36.291278 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:36.292184 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:36.292189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:36.292192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:36.292319 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:36.292860 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:36.292865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:36.292869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:36.293273 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:36.293431 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:36.294311 (+ 880us) server_negotiation.cc:299] Negotiation successful
0928 11:02:36.294552 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":210,"threads_started":1}
I20260928 11:02:36.295794 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:36.276237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36185 (local address 127.3.252.65:57477)
0928 11:02:36.277077 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:36.277141 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:36.278932 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:36.279614 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:36.279623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:36.280161 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:36.280738 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:36.280757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.282140 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.282144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:36.283585 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:36.283599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.284560 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.284569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:36.284840 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:36.285565 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:36.285596 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:36.288311 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:36.291424 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:36.291434 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:36.291455 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:36.292052 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:36.292421 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:36.292425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:36.292428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:36.292695 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:36.293491 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:36.293499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:36.293912 (+ 413us) client_negotiation.cc:769] Sending connection context
0928 11:02:36.294187 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 11:02:36.294629 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":342,"threads_started":1}
I20260928 11:02:36.297350 22533 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36185
I20260928 11:02:36.297644 22533 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:36.298396 22533 heartbeater.cc:511] Master 127.3.252.126:36185 requested a full tablet report, sending...
I20260928 11:02:36.300559 22344 ts_manager.cc:194] Registered new tserver with Master: 23ea5616bf154253a8d876f22d58c460 (127.3.252.65:41507)
I20260928 11:02:36.302052 22344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:57477
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 11:02:36.337108 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36185
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:36.653791 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:36.654117 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:36.654260 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:36.661412 22539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:36.661532 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:36.661693 22539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:02:36.671362 22539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:36.672655 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:36.673935 22539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:36.686585 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:36.686546 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:36.686626 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:36.687394 22539 server_base.cc:1034] running on GCE node
I20260928 11:02:36.688019 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:36.688984 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:36.690210 22539 hybrid_clock.cc:648] HybridClock initialized: now 1790593356690108 us; error 83 us; skew 500 ppm
I20260928 11:02:36.690559 22539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:36 dist-test-slave-djjw krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:36.694234 22539 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:02:36.695866 22539 webserver.cc:460] Webserver started at http://127.3.252.66:37017/ using document root <none> and password file <none>
I20260928 11:02:36.696659 22539 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:36.696777 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:36.697100 22539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:36.700866 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "5e38a36ca4f745c29afe75c9a1c320b8"
format_stamp: "Formatted at 2026-09-28 11:02:36 on dist-test-slave-djjw"
server_key: "c98114ee13e9db0c4d56606056a857a3"
server_key_iv: "c50a00f0540e5b6d45752b2a8cf35aae"
server_key_version: "encryptionkey@0"
I20260928 11:02:36.701870 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "5e38a36ca4f745c29afe75c9a1c320b8"
format_stamp: "Formatted at 2026-09-28 11:02:36 on dist-test-slave-djjw"
server_key: "c98114ee13e9db0c4d56606056a857a3"
server_key_iv: "c50a00f0540e5b6d45752b2a8cf35aae"
server_key_version: "encryptionkey@0"
I20260928 11:02:36.708474 22539 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 11:02:36.712219 22554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:36.713676 22539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 11:02:36.713896 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "5e38a36ca4f745c29afe75c9a1c320b8"
format_stamp: "Formatted at 2026-09-28 11:02:36 on dist-test-slave-djjw"
server_key: "c98114ee13e9db0c4d56606056a857a3"
server_key_iv: "c50a00f0540e5b6d45752b2a8cf35aae"
server_key_version: "encryptionkey@0"
I20260928 11:02:36.714116 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:36.729336 22539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:36.733565 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:36.733868 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:36.734843 22539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:36.736142 22539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:36.736218 22539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:36.736339 22539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:36.736392 22539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:36.765758 22539 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:36547
I20260928 11:02:36.765779 22666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:36547 every 8 connection(s)
I20260928 11:02:36.767076 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:02:36.769423 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 22539
I20260928 11:02:36.769599 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:02:36.770104 4081 external_mini_cluster.cc:1428] Setting key e3ab3ec439c3f126677c4a4a7c827d89
Sep 28 11:02:36 dist-test-slave-djjw krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:36.792136 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:36.769589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:42365 (local address 127.3.252.126:36185)
0928 11:02:36.769803 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:36.769809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:36.771043 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 11:02:36.773154 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:36.773158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:36.773225 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:36.773308 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:36.775196 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.775904 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.777932 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.778370 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.778471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:36.785587 (+ 7116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:36.785594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:36.785599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:36.785643 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:36.788669 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:36.789431 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:36.789436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:36.789440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:36.789597 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:36.790123 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:36.790129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:36.790132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:36.790488 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:36.790628 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:36.791527 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 11:02:36.791765 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 11:02:36.792949 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:36.770067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36185 (local address 127.3.252.66:42365)
0928 11:02:36.770828 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:36.770868 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:36.772857 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:36.773557 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:36.773569 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:36.774301 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:36.775019 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:36.775044 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.776049 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.776055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:36.777775 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:36.777790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:36.778577 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:36.778583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:36.778832 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:36.782971 (+ 4139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:36.782995 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:36.785299 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:36.788834 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:36.788841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:36.788857 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:36.789307 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:36.789719 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:36.789722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:36.789724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:36.790011 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:36.790602 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:36.790608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:36.791112 (+ 504us) client_negotiation.cc:769] Sending connection context
0928 11:02:36.791405 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 11:02:36.791811 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":283,"threads_started":1}
I20260928 11:02:36.794524 22667 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36185
I20260928 11:02:36.794809 22667 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
I20260928 11:02:36.795564 22667 heartbeater.cc:511] Master 127.3.252.126:36185 requested a full tablet report, sending...
I20260928 11:02:36.797089 22344 ts_manager.cc:194] Registered new tserver with Master: 5e38a36ca4f745c29afe75c9a1c320b8 (127.3.252.66:36547)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
I20260928 11:02:36.798043 22344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:42365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 11:02:36.825213 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36185
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:37.157001 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:37.157299 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:37.157419 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:37.164635 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:37.164762 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:37.164914 22672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:02:37.174935 22672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:37.176275 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:37.177600 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:37.191095 22678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:37.191155 22677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:37.191175 22680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:37.192011 22672 server_base.cc:1034] running on GCE node
I20260928 11:02:37.192632 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:37.193595 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:37.194806 22672 hybrid_clock.cc:648] HybridClock initialized: now 1790593357194732 us; error 69 us; skew 500 ppm
I20260928 11:02:37.195127 22672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:37 dist-test-slave-djjw krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593357, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:37.202126 22672 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:02:37.203716 22672 webserver.cc:460] Webserver started at http://127.3.252.67:33665/ using document root <none> and password file <none>
I20260928 11:02:37.204473 22672 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:37.204595 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:37.204896 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:37.207895 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "24062dcaae264fa68248f081fb9b5663"
format_stamp: "Formatted at 2026-09-28 11:02:37 on dist-test-slave-djjw"
server_key: "813157944dadc8987197902f685ce388"
server_key_iv: "dcea433961360ab645745e6b92e31ada"
server_key_version: "encryptionkey@0"
I20260928 11:02:37.208686 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "24062dcaae264fa68248f081fb9b5663"
format_stamp: "Formatted at 2026-09-28 11:02:37 on dist-test-slave-djjw"
server_key: "813157944dadc8987197902f685ce388"
server_key_iv: "dcea433961360ab645745e6b92e31ada"
server_key_version: "encryptionkey@0"
I20260928 11:02:37.214059 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260928 11:02:37.217707 22687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:37.219195 22672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:37.219362 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "24062dcaae264fa68248f081fb9b5663"
format_stamp: "Formatted at 2026-09-28 11:02:37 on dist-test-slave-djjw"
server_key: "813157944dadc8987197902f685ce388"
server_key_iv: "dcea433961360ab645745e6b92e31ada"
server_key_version: "encryptionkey@0"
I20260928 11:02:37.219583 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:37.240351 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:37.244585 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:37.244901 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:37.245913 22672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:37.247239 22672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:37.247315 22672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:37.247440 22672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:37.247496 22672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:37.282624 22672 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:43999
I20260928 11:02:37.282660 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:43999 every 8 connection(s)
I20260928 11:02:37.283820 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:02:37.288461 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 22672
I20260928 11:02:37.288662 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:02:37.289181 4081 external_mini_cluster.cc:1428] Setting key ab1b7dbe6787e2b25bbdba054276c9a2
Sep 28 11:02:37 dist-test-slave-djjw krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593357, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:37.304250 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.286293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:54857 (local address 127.3.252.126:36185)
0928 11:02:37.286453 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.286458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.287593 (+ 1135us) server_negotiation.cc:407] Connection header received
0928 11:02:37.289941 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.289947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.290074 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.290181 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:37.292137 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.292965 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.294576 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.294995 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.295105 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.298516 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:37.298521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:37.298524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:37.298559 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:37.300984 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:37.301751 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:37.301757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:37.301760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:37.301927 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:37.302345 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:37.302350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:37.302353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:37.302745 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:37.302895 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.303667 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.303893 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 11:02:37.305083 22533 heartbeater.cc:503] Master 127.3.252.126:36185 was elected leader, sending a full tablet report...
I20260928 11:02:37.305539 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.286717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36185 (local address 127.3.252.67:54857)
0928 11:02:37.287423 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:37.287461 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:37.289607 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:37.290443 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:37.290451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:37.291096 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:37.291958 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.291986 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.293116 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.293120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.294434 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.294447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.295229 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.295236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.295464 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.296077 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:37.296105 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:37.298294 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:37.301126 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:37.301144 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:37.301161 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:37.301632 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:37.302029 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:37.302034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:37.302038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:37.302233 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:37.302868 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:37.302877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:37.303254 (+ 377us) client_negotiation.cc:769] Sending connection context
0928 11:02:37.303569 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 11:02:37.304104 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":267,"threads_started":1}
I20260928 11:02:37.306959 22800 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36185
I20260928 11:02:37.307219 22800 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:37.307880 22800 heartbeater.cc:511] Master 127.3.252.126:36185 requested a full tablet report, sending...
I20260928 11:02:37.309244 22344 ts_manager.cc:194] Registered new tserver with Master: 24062dcaae264fa68248f081fb9b5663 (127.3.252.67:43999)
I20260928 11:02:37.310084 22344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:54857
I20260928 11:02:37.316946 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:02:37.333947 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.321035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.3.252.126:36185)
0928 11:02:37.321251 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.321259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.321667 (+ 408us) server_negotiation.cc:407] Connection header received
0928 11:02:37.321802 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.321806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.321919 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.321998 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:37.323483 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.324319 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.325682 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.326170 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.326276 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.328780 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:37.328785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:37.328788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:37.328824 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:37.331244 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:37.331942 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:37.331946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:37.331949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:37.332113 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:37.332505 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:37.332509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:37.332512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:37.332838 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:37.333073 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.333398 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.333573 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 11:02:37.338059 22344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:02:37.344801 22344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:02:37.378564 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.356838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.3.252.65:41507)
0928 11:02:37.370712 (+ 13874us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.370720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.370748 (+ 28us) server_negotiation.cc:407] Connection header received
0928 11:02:37.370843 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.370851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.371134 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.371467 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:37.372074 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.373473 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.376229 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.377403 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.377538 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.377776 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.378079 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.378243 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13704,"thread_start_us":13554,"threads_started":1}
I20260928 11:02:37.378688 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.356815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:41507 (local address 127.0.0.1:57306)
0928 11:02:37.369099 (+ 12284us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:37.369141 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:37.369324 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:37.371326 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:37.371332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:37.371382 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:37.371871 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.371885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.373611 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.373617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.376067 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.376084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.377596 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.377602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.377802 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.377886 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 11:02:37.378024 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 11:02:37.378214 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12001,"mutex_wait_us":16,"spinlock_wait_cycles":9728,"thread_start_us":13758,"threads_started":1}
I20260928 11:02:37.379652 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.370950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:36547 (local address 127.0.0.1:41922)
0928 11:02:37.372450 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:37.372465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:37.372591 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:37.373168 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:37.373174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:37.373200 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:37.373517 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.373528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.375008 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.375013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.377035 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.377045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.378889 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.378896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.379089 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.379122 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 11:02:37.379206 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 11:02:37.379327 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"thread_start_us":1198,"threads_started":1}
I20260928 11:02:37.379731 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.371254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.3.252.66:36547)
0928 11:02:37.372075 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.372083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.372568 (+ 485us) server_negotiation.cc:407] Connection header received
0928 11:02:37.372753 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.372760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.372971 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.373173 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:37.373664 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.374827 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.377171 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.378431 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.378561 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.378737 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.379275 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.379431 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":519,"threads_started":1}
I20260928 11:02:37.380090 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.356860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:43999 (local address 127.0.0.1:59890)
0928 11:02:37.370024 (+ 13164us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:37.370046 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:37.370188 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:37.371323 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:37.371331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:37.371383 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:37.372560 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.372573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.374460 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.374466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.376729 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.376747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.379001 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.379011 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.379182 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.379214 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 11:02:37.379297 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 11:02:37.379435 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12899,"mutex_wait_us":5,"spinlock_wait_cycles":6016,"thread_start_us":11794,"threads_started":1}
I20260928 11:02:37.380908 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.357629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.3.252.67:43999)
0928 11:02:37.369891 (+ 12262us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.369900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.370320 (+ 420us) server_negotiation.cc:407] Connection header received
0928 11:02:37.370447 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.370477 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.370778 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.370992 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:37.372775 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.374317 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.376906 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.378204 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.378323 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.378538 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.380445 (+ 1907us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.380611 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12044,"thread_start_us":12356,"threads_started":1}
I20260928 11:02:37.383018 22602 tablet_service.cc:1467] Processing CreateTablet for tablet 67cbe2289eae44b694d117438a44acb4 (DEFAULT_TABLE table=test-table [id=6b3628ccb28b474ebcad77c27c2a307d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:37.383019 22468 tablet_service.cc:1467] Processing CreateTablet for tablet 67cbe2289eae44b694d117438a44acb4 (DEFAULT_TABLE table=test-table [id=6b3628ccb28b474ebcad77c27c2a307d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:37.384191 22602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67cbe2289eae44b694d117438a44acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:37.384363 22468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67cbe2289eae44b694d117438a44acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:37.384111 22735 tablet_service.cc:1467] Processing CreateTablet for tablet 67cbe2289eae44b694d117438a44acb4 (DEFAULT_TABLE table=test-table [id=6b3628ccb28b474ebcad77c27c2a307d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:37.385469 22735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67cbe2289eae44b694d117438a44acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:37.395414 22818 tablet_bootstrap.cc:492] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Bootstrap starting.
I20260928 11:02:37.398872 22818 tablet_bootstrap.cc:654] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:37.399394 22819 tablet_bootstrap.cc:492] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Bootstrap starting.
I20260928 11:02:37.400364 22818 log.cc:824] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:37.401728 22820 tablet_bootstrap.cc:492] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Bootstrap starting.
I20260928 11:02:37.402890 22819 tablet_bootstrap.cc:654] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:37.404189 22819 log.cc:824] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:37.405077 22820 tablet_bootstrap.cc:654] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:37.406147 22818 tablet_bootstrap.cc:492] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: No bootstrap required, opened a new log
I20260928 11:02:37.406205 22820 log.cc:824] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:37.406409 22818 ts_tablet_manager.cc:1397] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 11:02:37.407133 22819 tablet_bootstrap.cc:492] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: No bootstrap required, opened a new log
I20260928 11:02:37.407357 22819 ts_tablet_manager.cc:1397] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 11:02:37.409076 22820 tablet_bootstrap.cc:492] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: No bootstrap required, opened a new log
I20260928 11:02:37.409325 22820 ts_tablet_manager.cc:1397] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 11:02:37.413923 22818 raft_consensus.cc:348] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.414412 22818 raft_consensus.cc:374] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:37.414523 22818 raft_consensus.cc:729] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e38a36ca4f745c29afe75c9a1c320b8, State: Initialized, Role: FOLLOWER
I20260928 11:02:37.415125 22818 consensus_queue.cc:260] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.416219 22667 heartbeater.cc:503] Master 127.3.252.126:36185 was elected leader, sending a full tablet report...
I20260928 11:02:37.418124 22818 ts_tablet_manager.cc:1428] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260928 11:02:37.418944 22819 raft_consensus.cc:348] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.419340 22819 raft_consensus.cc:374] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:37.419433 22819 raft_consensus.cc:729] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ea5616bf154253a8d876f22d58c460, State: Initialized, Role: FOLLOWER
I20260928 11:02:37.419292 22820 raft_consensus.cc:348] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.420125 22820 raft_consensus.cc:374] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:37.420022 22819 consensus_queue.cc:260] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.420228 22820 raft_consensus.cc:729] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24062dcaae264fa68248f081fb9b5663, State: Initialized, Role: FOLLOWER
I20260928 11:02:37.420864 22820 consensus_queue.cc:260] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.422672 22819 ts_tablet_manager.cc:1428] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 11:02:37.424328 22800 heartbeater.cc:503] Master 127.3.252.126:36185 was elected leader, sending a full tablet report...
I20260928 11:02:37.424978 22820 ts_tablet_manager.cc:1428] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
W20260928 11:02:37.518921 22668 tablet.cc:2369] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 11:02:37.525318 22534 tablet.cc:2369] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 11:02:37.535523 22801 tablet.cc:2369] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:02:37.755493 22824 raft_consensus.cc:482] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:37.755805 22824 raft_consensus.cc:504] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.757510 22824 leader_election.cc:288] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ea5616bf154253a8d876f22d58c460 (127.3.252.65:41507), 24062dcaae264fa68248f081fb9b5663 (127.3.252.67:43999)
I20260928 11:02:37.765981 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.758155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:46385 (local address 127.3.252.65:41507)
0928 11:02:37.758543 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.758549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.758721 (+ 172us) server_negotiation.cc:407] Connection header received
0928 11:02:37.758932 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.758936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.759015 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.759103 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:37.759698 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.760827 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.763652 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.764884 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.765049 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.765132 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.765587 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.765721 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260928 11:02:37.766144 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.757803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:41507 (local address 127.3.252.66:46385)
0928 11:02:37.758620 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:37.758638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:37.758794 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:37.759147 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:37.759151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:37.759176 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:37.759550 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.759562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.760994 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.761000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.763475 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.763491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.765167 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.765172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.765397 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.765440 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 11:02:37.765538 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 11:02:37.765697 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":91,"thread_start_us":463,"threads_started":1}
I20260928 11:02:37.766160 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.758093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:44075 (local address 127.3.252.67:43999)
0928 11:02:37.758379 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.758386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.759031 (+ 645us) server_negotiation.cc:407] Connection header received
0928 11:02:37.759176 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.759182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.759256 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.759326 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:37.759903 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.760896 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.763653 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.764883 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.765665 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.765735 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.765785 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.765936 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 11:02:37.766724 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.757909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:43999 (local address 127.3.252.66:44075)
0928 11:02:37.758923 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:37.758937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:37.759053 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:37.759376 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:37.759380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:37.759401 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:37.759762 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.759770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.761023 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.761027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.763474 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:37.763491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.765161 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.765168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:37.765429 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.765455 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 11:02:37.765530 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 11:02:37.766439 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":821,"thread_start_us":753,"threads_started":1}
I20260928 11:02:37.767422 22488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe2289eae44b694d117438a44acb4" candidate_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ea5616bf154253a8d876f22d58c460" is_pre_election: true
I20260928 11:02:37.767666 22755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe2289eae44b694d117438a44acb4" candidate_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24062dcaae264fa68248f081fb9b5663" is_pre_election: true
I20260928 11:02:37.767979 22488 raft_consensus.cc:2454] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e38a36ca4f745c29afe75c9a1c320b8 in term 0.
I20260928 11:02:37.768095 22755 raft_consensus.cc:2454] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e38a36ca4f745c29afe75c9a1c320b8 in term 0.
I20260928 11:02:37.768664 22557 leader_election.cc:302] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ea5616bf154253a8d876f22d58c460, 5e38a36ca4f745c29afe75c9a1c320b8; no voters:
I20260928 11:02:37.769052 22824 raft_consensus.cc:2781] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:02:37.769199 22824 raft_consensus.cc:482] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:02:37.769306 22824 raft_consensus.cc:3037] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:37.772003 22824 raft_consensus.cc:504] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.772627 22824 leader_election.cc:288] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [CANDIDATE]: Term 1 election: Requested vote from peers 23ea5616bf154253a8d876f22d58c460 (127.3.252.65:41507), 24062dcaae264fa68248f081fb9b5663 (127.3.252.67:43999)
I20260928 11:02:37.773057 22488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe2289eae44b694d117438a44acb4" candidate_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ea5616bf154253a8d876f22d58c460"
I20260928 11:02:37.773177 22755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe2289eae44b694d117438a44acb4" candidate_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24062dcaae264fa68248f081fb9b5663"
I20260928 11:02:37.773281 22488 raft_consensus.cc:3037] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:37.773371 22755 raft_consensus.cc:3037] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:37.776410 22755 raft_consensus.cc:2454] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e38a36ca4f745c29afe75c9a1c320b8 in term 1.
I20260928 11:02:37.776439 22488 raft_consensus.cc:2454] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e38a36ca4f745c29afe75c9a1c320b8 in term 1.
I20260928 11:02:37.776794 22557 leader_election.cc:302] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ea5616bf154253a8d876f22d58c460, 5e38a36ca4f745c29afe75c9a1c320b8; no voters:
I20260928 11:02:37.777110 22824 raft_consensus.cc:2781] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:37.777570 22824 raft_consensus.cc:686] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 1 LEADER]: Becoming Leader. State: Replica: 5e38a36ca4f745c29afe75c9a1c320b8, State: Running, Role: LEADER
I20260928 11:02:37.778111 22824 consensus_queue.cc:237] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } }
I20260928 11:02:37.782388 22343 catalog_manager.cc:5495] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e38a36ca4f745c29afe75c9a1c320b8 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e38a36ca4f745c29afe75c9a1c320b8" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36547 } health_report { overall_health: HEALTHY } } }
I20260928 11:02:37.826767 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.821472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.3.252.66:36547)
0928 11:02:37.821657 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.821662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.821680 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:02:37.821761 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.821764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.821882 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.822037 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:02:37.822406 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.823215 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.824622 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.825044 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.825145 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.825647 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:02:37.826206 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:02:37.826333 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.826514 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.826558 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 11:02:37.834893 22488 raft_consensus.cc:1261] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 1 FOLLOWER]: Refusing update from remote peer 5e38a36ca4f745c29afe75c9a1c320b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:37.834895 22755 raft_consensus.cc:1261] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 1 FOLLOWER]: Refusing update from remote peer 5e38a36ca4f745c29afe75c9a1c320b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:37.835963 22824 consensus_queue.cc:1035] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24062dcaae264fa68248f081fb9b5663" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:37.836279 22829 consensus_queue.cc:1035] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ea5616bf154253a8d876f22d58c460" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:37.862222 22344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:37.862479 22344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:37.865751 22344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7c4b06d4a2d4e878492a111ab121dc8: Sending DeleteTablet for 3 replicas of tablet 67cbe2289eae44b694d117438a44acb4
I20260928 11:02:37.866967 22735 tablet_service.cc:1514] Processing DeleteTablet for tablet 67cbe2289eae44b694d117438a44acb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:37 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:59890
I20260928 11:02:37.867000 22602 tablet_service.cc:1514] Processing DeleteTablet for tablet 67cbe2289eae44b694d117438a44acb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:37 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:41922
I20260928 11:02:37.867678 22468 tablet_service.cc:1514] Processing DeleteTablet for tablet 67cbe2289eae44b694d117438a44acb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:37 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:57306
I20260928 11:02:37.870543 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 22405
I20260928 11:02:37.878376 22842 tablet_replica.cc:330] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: stopping tablet replica
I20260928 11:02:37.878675 22844 tablet_replica.cc:330] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: stopping tablet replica
I20260928 11:02:37.878866 22842 raft_consensus.cc:2229] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:37.879347 22843 tablet_replica.cc:330] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: stopping tablet replica
I20260928 11:02:37.879411 22842 raft_consensus.cc:2258] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:37.879583 22844 raft_consensus.cc:2229] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:37.879725 22843 raft_consensus.cc:2229] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:02:37.880123 22844 raft_consensus.cc:2258] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:37.880359 22843 raft_consensus.cc:2258] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:37.881876 22842 ts_tablet_manager.cc:1905] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:37.882459 22844 ts_tablet_manager.cc:1905] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:37.888864 22842 ts_tablet_manager.cc:1918] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:37.888955 22844 ts_tablet_manager.cc:1918] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:37.889056 22842 log.cc:1196] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/67cbe2289eae44b694d117438a44acb4
Sep 28 11:02:37 dist-test-slave-djjw krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 11:02:37.889103 22844 log.cc:1196] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/67cbe2289eae44b694d117438a44acb4
I20260928 11:02:37.889580 22842 ts_tablet_manager.cc:1939] T 67cbe2289eae44b694d117438a44acb4 P 24062dcaae264fa68248f081fb9b5663: Deleting consensus metadata
I20260928 11:02:37.889599 22844 ts_tablet_manager.cc:1939] T 67cbe2289eae44b694d117438a44acb4 P 23ea5616bf154253a8d876f22d58c460: Deleting consensus metadata
I20260928 11:02:37.891044 22843 ts_tablet_manager.cc:1905] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:37.891402 22331 catalog_manager.cc:4841] TS 24062dcaae264fa68248f081fb9b5663 (127.3.252.67:43999): tablet 67cbe2289eae44b694d117438a44acb4 (table test-table [id=6b3628ccb28b474ebcad77c27c2a307d]) successfully deleted
I20260928 11:02:37.891402 22330 catalog_manager.cc:4841] TS 23ea5616bf154253a8d876f22d58c460 (127.3.252.65:41507): tablet 67cbe2289eae44b694d117438a44acb4 (table test-table [id=6b3628ccb28b474ebcad77c27c2a307d]) successfully deleted
I20260928 11:02:37.896567 22843 ts_tablet_manager.cc:1918] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:37.896742 22843 log.cc:1196] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/67cbe2289eae44b694d117438a44acb4
I20260928 11:02:37.897243 22843 ts_tablet_manager.cc:1939] T 67cbe2289eae44b694d117438a44acb4 P 5e38a36ca4f745c29afe75c9a1c320b8: Deleting consensus metadata
I20260928 11:02:37.898769 22330 catalog_manager.cc:4841] TS 5e38a36ca4f745c29afe75c9a1c320b8 (127.3.252.66:36547): tablet 67cbe2289eae44b694d117438a44acb4 (table test-table [id=6b3628ccb28b474ebcad77c27c2a307d]) successfully deleted
I20260928 11:02:37.899484 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:37.871679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.3.252.65:41507)
0928 11:02:37.871863 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:37.871869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:37.878782 (+ 6913us) server_negotiation.cc:407] Connection header received
0928 11:02:37.878977 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:37.878984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:37.879065 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:37.879158 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:37.881414 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.882492 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.884221 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:37.884836 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:37.884925 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:37.892582 (+ 7657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:37.892596 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:37.892614 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:37.892657 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:37.896387 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:37.897251 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:37.897260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:37.897264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:37.897419 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:37.897815 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:37.897820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:37.897823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:37.898307 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:37.898598 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:37.898891 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 11:02:37.899123 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 11:02:37.900143 22528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:37.988637 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 22405
I20260928 11:02:38.003674 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 22539
Sep 28 11:02:38 dist-test-slave-djjw krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 11:02:38.020542 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:38.004504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.3.252.66:36547)
0928 11:02:38.004703 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:38.004710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:38.004731 (+ 21us) server_negotiation.cc:407] Connection header received
0928 11:02:38.004796 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:38.004801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:38.004886 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:38.004998 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:38.006218 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:38.007424 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:38.008670 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:38.009275 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:38.009403 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:38.013438 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:38.013459 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:38.013481 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:38.013586 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:38.017494 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:38.018281 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:38.018290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:38.018298 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:38.018468 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:38.018839 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:38.018844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:38.018847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:38.019307 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:38.019638 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:38.019901 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 11:02:38.020130 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 11:02:38.021263 22662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:38.135778 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 22539
I20260928 11:02:38.151609 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 22672
Sep 28 11:02:38 dist-test-slave-djjw krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 11:02:38.166209 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:38.152348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.3.252.67:43999)
0928 11:02:38.152539 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:38.152546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:38.152565 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:02:38.152715 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:38.152718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:38.152783 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:38.152866 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:38.154238 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:38.154952 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:38.156228 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:38.156638 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:38.156747 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:38.160829 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:38.160844 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:38.160860 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:38.160903 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:38.163514 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:38.164191 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:38.164199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:38.164202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:38.164344 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:38.164688 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:38.164692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:38.164695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:38.165056 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:38.165327 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:38.165648 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 11:02:38.165892 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 11:02:38.166834 22795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:38.255141 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 22672
I20260928 11:02:38.268980 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 22312
I20260928 11:02:38.270067 22374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:38.379040 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 22312
2026-09-28T11:02:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4402 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T11:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:38Z Disabled control of system clock
I20260928 11:02:38.450423 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33393
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:33393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 11:02:38.772786 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:38.773075 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:38.773175 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:38.780323 22853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:38.780436 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:38.780483 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:38.780560 22853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:38.780612 22853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:38.789907 22853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:33393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:38.791181 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:38.793023 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:38.804427 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:38.804661 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:38.804666 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:38.805534 22853 server_base.cc:1034] running on GCE node
I20260928 11:02:38.806286 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:38.807659 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:38.808863 22853 hybrid_clock.cc:648] HybridClock initialized: now 1790593358808811 us; error 45 us; skew 500 ppm
I20260928 11:02:38.809202 22853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:38.811300 22853 webserver.cc:460] Webserver started at http://127.3.252.126:33707/ using document root <none> and password file <none>
I20260928 11:02:38.811944 22853 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:38.812042 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:38.812325 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:38.815081 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "c8498e7c875f4c8dbef7bfb9d9532796"
format_stamp: "Formatted at 2026-09-28 11:02:38 on dist-test-slave-djjw"
server_key: "8b61998852ba0966b847a24fa06caf58"
server_key_iv: "7baca9992e8c14aafd81f70b58629c7c"
server_key_version: "encryptionkey@0"
I20260928 11:02:38.815807 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "c8498e7c875f4c8dbef7bfb9d9532796"
format_stamp: "Formatted at 2026-09-28 11:02:38 on dist-test-slave-djjw"
server_key: "8b61998852ba0966b847a24fa06caf58"
server_key_iv: "7baca9992e8c14aafd81f70b58629c7c"
server_key_version: "encryptionkey@0"
I20260928 11:02:38.820600 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 11:02:38.823839 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:38.825105 22853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:02:38.825264 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "c8498e7c875f4c8dbef7bfb9d9532796"
format_stamp: "Formatted at 2026-09-28 11:02:38 on dist-test-slave-djjw"
server_key: "8b61998852ba0966b847a24fa06caf58"
server_key_iv: "7baca9992e8c14aafd81f70b58629c7c"
server_key_version: "encryptionkey@0"
I20260928 11:02:38.825459 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:38.841109 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:38.841946 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:38.842197 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:38.858899 22853 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:33393
I20260928 11:02:38.858922 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:33393 every 8 connection(s)
I20260928 11:02:38.860314 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:38.863500 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 22853
I20260928 11:02:38.863669 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:38.864192 4081 external_mini_cluster.cc:1428] Setting key a14bb3a27890234c926d88658a468572
I20260928 11:02:38.864389 22919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:38.878010 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:38.866115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.3.252.126:33393)
0928 11:02:38.867111 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:38.867123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:38.867167 (+ 44us) server_negotiation.cc:407] Connection header received
0928 11:02:38.869107 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:38.869131 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:38.869618 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:38.870003 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:38.870568 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:38.871895 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:38.873670 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:38.874406 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:38.874555 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:38.875776 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:38.875783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:38.875801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:38.876349 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:38.876423 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:38.876443 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:38.876526 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:38.876731 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 11:02:38.877063 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":348,"threads_started":1}
I20260928 11:02:38.878181 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796: Bootstrap starting.
I20260928 11:02:38.881134 22919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:38.882094 22919 log.cc:824] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:38.884725 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796: No bootstrap required, opened a new log
I20260928 11:02:38.891896 22919 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33393 } }
I20260928 11:02:38.892256 22919 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:38.892355 22919 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8498e7c875f4c8dbef7bfb9d9532796, State: Initialized, Role: FOLLOWER
I20260928 11:02:38.892877 22919 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33393 } }
I20260928 11:02:38.893090 22919 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:38.893213 22919 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:38.893364 22919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:38.895526 22919 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33393 } }
I20260928 11:02:38.896103 22919 leader_election.cc:302] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8498e7c875f4c8dbef7bfb9d9532796; no voters:
I20260928 11:02:38.896644 22919 leader_election.cc:288] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:38.896763 22924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:38.897436 22924 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [term 1 LEADER]: Becoming Leader. State: Replica: c8498e7c875f4c8dbef7bfb9d9532796, State: Running, Role: LEADER
I20260928 11:02:38.897995 22924 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33393 } }
I20260928 11:02:38.898597 22919 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:38.901298 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8498e7c875f4c8dbef7bfb9d9532796. Latest consensus state: current_term: 1 leader_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33393 } } }
I20260928 11:02:38.902485 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:38.901536 22925 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8498e7c875f4c8dbef7bfb9d9532796" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 33393 } } }
I20260928 11:02:38.903422 22925 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:38.904868 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:38.909740 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:38.918679 22933 catalog_manager.cc:1347] Generated new cluster ID: ad2f9e467d344fbdbed38722dd09c071
I20260928 11:02:38.918804 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:38.932008 22933 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:38.933393 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:38.944002 22933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796: Generated new TSK 0
I20260928 11:02:38.944810 22933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:38.949956 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33393
--builtin_ntp_servers=127.3.252.84:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 11:02:39.279501 22943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:39.279796 22943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:39.279904 22943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:39.287295 22943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:39.287402 22943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:39.287544 22943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:02:39.296950 22943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:39.298346 22943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:39.300213 22943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:39.313185 22949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:39.313210 22951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:39.313118 22948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:39.313802 22943 server_base.cc:1034] running on GCE node
I20260928 11:02:39.314483 22943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:39.315512 22943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:39.316740 22943 hybrid_clock.cc:648] HybridClock initialized: now 1790593359316674 us; error 61 us; skew 500 ppm
I20260928 11:02:39.317068 22943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:39.319590 22943 webserver.cc:460] Webserver started at http://127.3.252.65:36799/ using document root <none> and password file <none>
I20260928 11:02:39.320338 22943 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:39.320533 22943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:39.320906 22943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:39.324019 22943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "61e1ed0a75364241a27a558b2fb03586"
format_stamp: "Formatted at 2026-09-28 11:02:39 on dist-test-slave-djjw"
server_key: "3932ab13f671074096ef7ed6e33fdf59"
server_key_iv: "d2c894a10a20b05ee79864469d6d1fe5"
server_key_version: "encryptionkey@0"
I20260928 11:02:39.324826 22943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "61e1ed0a75364241a27a558b2fb03586"
format_stamp: "Formatted at 2026-09-28 11:02:39 on dist-test-slave-djjw"
server_key: "3932ab13f671074096ef7ed6e33fdf59"
server_key_iv: "d2c894a10a20b05ee79864469d6d1fe5"
server_key_version: "encryptionkey@0"
I20260928 11:02:39.330120 22943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 11:02:39.333735 22957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:39.335301 22943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:39.335465 22943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "61e1ed0a75364241a27a558b2fb03586"
format_stamp: "Formatted at 2026-09-28 11:02:39 on dist-test-slave-djjw"
server_key: "3932ab13f671074096ef7ed6e33fdf59"
server_key_iv: "d2c894a10a20b05ee79864469d6d1fe5"
server_key_version: "encryptionkey@0"
I20260928 11:02:39.335690 22943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:39.351649 22943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:39.352504 22943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:39.352882 22943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:39.353847 22943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:39.355340 22943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:39.355423 22943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:39.355525 22943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:39.355598 22943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:39.384891 22943 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:39451
I20260928 11:02:39.384939 23069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:39451 every 8 connection(s)
I20260928 11:02:39.386190 22943 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:02:39.396476 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 22943
I20260928 11:02:39.396620 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:02:39.397078 4081 external_mini_cluster.cc:1428] Setting key 13188139dc5b2d6abcc554fcc915f573
I20260928 11:02:39.398535 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:39.388747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:37815 (local address 127.3.252.126:33393)
0928 11:02:39.389223 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:39.389228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:39.390249 (+ 1021us) server_negotiation.cc:407] Connection header received
0928 11:02:39.392014 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:39.392021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:39.392101 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:39.392195 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:39.393269 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.394034 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.395547 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.395953 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.396066 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:39.397397 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:39.397403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:39.397406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:39.397424 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:39.397468 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:39.397480 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:39.397559 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:39.398088 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 11:02:39.398187 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
I20260928 11:02:39.399667 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:39.389197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33393 (local address 127.3.252.65:37815)
0928 11:02:39.390096 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:39.390132 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:39.391741 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:39.392446 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:39.392455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:39.392551 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:39.393100 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:39.393117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.394177 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.394181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:39.395406 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:39.395418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.396189 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.396197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:39.396414 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:39.397076 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:39.397079 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:39.397119 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:39.397219 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:39.397870 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:39.397882 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:39.397891 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 11:02:39.398055 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 11:02:39.398517 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":261,"threads_started":1}
I20260928 11:02:39.399987 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33393
--builtin_ntp_servers=127.3.252.84:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 11:02:39.401229 23070 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33393
I20260928 11:02:39.401512 23070 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:39.402262 23070 heartbeater.cc:511] Master 127.3.252.126:33393 requested a full tablet report, sending...
I20260928 11:02:39.404398 22884 ts_manager.cc:194] Registered new tserver with Master: 61e1ed0a75364241a27a558b2fb03586 (127.3.252.65:39451)
I20260928 11:02:39.406376 22884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.65:37815
W20260928 11:02:39.724116 23074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:39.724431 23074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:39.724524 23074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:39.732224 23074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:39.732348 23074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:39.732457 23074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:02:39.741696 23074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:39.742995 23074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:39.744771 23074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:39.757290 23079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:39.757187 23082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:39.757303 23080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:39.758496 23074 server_base.cc:1034] running on GCE node
I20260928 11:02:39.759192 23074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:39.760151 23074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:39.761495 23074 hybrid_clock.cc:648] HybridClock initialized: now 1790593359761452 us; error 29 us; skew 500 ppm
I20260928 11:02:39.761871 23074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:39.764107 23074 webserver.cc:460] Webserver started at http://127.3.252.66:45843/ using document root <none> and password file <none>
I20260928 11:02:39.764811 23074 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:39.764909 23074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:39.765221 23074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:39.768113 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "ab00aad45259413c8c9cc7fb8e2eabfc"
format_stamp: "Formatted at 2026-09-28 11:02:39 on dist-test-slave-djjw"
server_key: "2517576295d9b54b6b0b2d608d45732f"
server_key_iv: "eb5d39287caabae96e2d113fef02de79"
server_key_version: "encryptionkey@0"
I20260928 11:02:39.768931 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "ab00aad45259413c8c9cc7fb8e2eabfc"
format_stamp: "Formatted at 2026-09-28 11:02:39 on dist-test-slave-djjw"
server_key: "2517576295d9b54b6b0b2d608d45732f"
server_key_iv: "eb5d39287caabae96e2d113fef02de79"
server_key_version: "encryptionkey@0"
I20260928 11:02:39.774089 23074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 11:02:39.777735 23088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:39.779174 23074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:39.779337 23074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "ab00aad45259413c8c9cc7fb8e2eabfc"
format_stamp: "Formatted at 2026-09-28 11:02:39 on dist-test-slave-djjw"
server_key: "2517576295d9b54b6b0b2d608d45732f"
server_key_iv: "eb5d39287caabae96e2d113fef02de79"
server_key_version: "encryptionkey@0"
I20260928 11:02:39.779546 23074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:39.811211 23074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:39.812040 23074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:39.812326 23074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:39.813287 23074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:39.814785 23074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:39.814884 23074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:39.814988 23074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:39.815054 23074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:39.845741 23074 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:32775
I20260928 11:02:39.845799 23200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:32775 every 8 connection(s)
I20260928 11:02:39.847105 23074 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:02:39.855721 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 23074
I20260928 11:02:39.855893 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:02:39.856346 4081 external_mini_cluster.cc:1428] Setting key 0f3d7d48bff39f614121074aa76f5905
I20260928 11:02:39.859522 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33393
--builtin_ntp_servers=127.3.252.84:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 11:02:39.859858 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:39.849722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:36051 (local address 127.3.252.126:33393)
0928 11:02:39.849979 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:39.849984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:39.851381 (+ 1397us) server_negotiation.cc:407] Connection header received
0928 11:02:39.853113 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:39.853118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:39.853192 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:39.853285 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:39.854421 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.855144 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.856825 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.857274 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.857397 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:39.858987 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:39.858994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:39.858998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:39.859021 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:39.859062 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:39.859080 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:39.859183 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:39.859430 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 11:02:39.859571 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 11:02:39.861086 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:39.850276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33393 (local address 127.3.252.66:36051)
0928 11:02:39.851205 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:39.851251 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:39.852834 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:39.853473 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:39.853481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:39.853572 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:39.854243 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:39.854261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.855279 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.855283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:39.856666 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:39.856678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:39.857481 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:39.857486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:39.857705 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:39.858588 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:39.858602 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:39.858660 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:39.858763 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:39.859193 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:39.859205 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:39.859213 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 11:02:39.859379 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 11:02:39.859801 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":272,"threads_started":1}
I20260928 11:02:39.867313 23201 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33393
I20260928 11:02:39.867651 23201 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:39.868590 23201 heartbeater.cc:511] Master 127.3.252.126:33393 requested a full tablet report, sending...
I20260928 11:02:39.870301 22884 ts_manager.cc:194] Registered new tserver with Master: ab00aad45259413c8c9cc7fb8e2eabfc (127.3.252.66:32775)
I20260928 11:02:39.871331 22884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.66:36051
W20260928 11:02:40.184916 23204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:40.185187 23204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:40.185295 23204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:40.192456 23204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:40.192564 23204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:40.192703 23204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:02:40.201750 23204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:33393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:40.203016 23204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:40.204761 23204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:40.217175 23210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:40.217176 23209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:40.217200 23212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:40.217773 23204 server_base.cc:1034] running on GCE node
I20260928 11:02:40.218489 23204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:40.219390 23204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:40.220613 23204 hybrid_clock.cc:648] HybridClock initialized: now 1790593360220570 us; error 35 us; skew 500 ppm
I20260928 11:02:40.220922 23204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:40.223270 23204 webserver.cc:460] Webserver started at http://127.3.252.67:36245/ using document root <none> and password file <none>
I20260928 11:02:40.223937 23204 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:40.224028 23204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:40.224319 23204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:40.227162 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "4567f1c1347e4f37a04d403cb1435284"
format_stamp: "Formatted at 2026-09-28 11:02:40 on dist-test-slave-djjw"
server_key: "6fbb07277f74e1690871d274c4ee2b1f"
server_key_iv: "572bc483f74967945a2dc98993cfdc43"
server_key_version: "encryptionkey@0"
I20260928 11:02:40.227877 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "4567f1c1347e4f37a04d403cb1435284"
format_stamp: "Formatted at 2026-09-28 11:02:40 on dist-test-slave-djjw"
server_key: "6fbb07277f74e1690871d274c4ee2b1f"
server_key_iv: "572bc483f74967945a2dc98993cfdc43"
server_key_version: "encryptionkey@0"
I20260928 11:02:40.232919 23204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 11:02:40.236719 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:40.238248 23204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:40.238405 23204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "4567f1c1347e4f37a04d403cb1435284"
format_stamp: "Formatted at 2026-09-28 11:02:40 on dist-test-slave-djjw"
server_key: "6fbb07277f74e1690871d274c4ee2b1f"
server_key_iv: "572bc483f74967945a2dc98993cfdc43"
server_key_version: "encryptionkey@0"
I20260928 11:02:40.238611 23204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:40.251072 23204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:40.251919 23204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:40.252207 23204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:40.253183 23204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:40.254781 23204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:40.254868 23204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:40.254979 23204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:40.255053 23204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:40.286393 23204 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:40251
I20260928 11:02:40.286451 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:40251 every 8 connection(s)
I20260928 11:02:40.287681 23204 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:02:40.295680 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 23204
I20260928 11:02:40.295855 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:02:40.296432 4081 external_mini_cluster.cc:1428] Setting key 45912d0d555ecb43225bf85eeec40135
I20260928 11:02:40.300835 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.290158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:54185 (local address 127.3.252.126:33393)
0928 11:02:40.290396 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.290402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.291894 (+ 1492us) server_negotiation.cc:407] Connection header received
0928 11:02:40.293673 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.293677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.293753 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.293876 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:40.295089 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.295871 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.297950 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.298505 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.298619 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.300140 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:40.300145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:40.300148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:40.300166 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:40.300198 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:40.300209 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:40.300278 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.300502 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.300589 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 11:02:40.301786 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.290726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:33393 (local address 127.3.252.67:54185)
0928 11:02:40.291714 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.291758 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:40.293386 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:40.294154 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:40.294164 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:40.294260 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:40.294876 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.294894 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.296039 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.296046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.297776 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.297793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.298781 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.298790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.299082 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.299820 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:40.299822 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:40.299865 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:40.299949 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:40.300283 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:40.300289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:40.300294 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 11:02:40.300428 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 11:02:40.300762 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":335,"threads_started":1}
I20260928 11:02:40.303131 23331 heartbeater.cc:347] Connected to a master server at 127.3.252.126:33393
I20260928 11:02:40.303412 23331 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:40.304203 23331 heartbeater.cc:511] Master 127.3.252.126:33393 requested a full tablet report, sending...
I20260928 11:02:40.305608 22884 ts_manager.cc:194] Registered new tserver with Master: 4567f1c1347e4f37a04d403cb1435284 (127.3.252.67:40251)
I20260928 11:02:40.306445 22884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.67:54185
I20260928 11:02:40.311923 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:02:40.322074 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.315905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.3.252.126:33393)
0928 11:02:40.316075 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.316081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.316630 (+ 549us) server_negotiation.cc:407] Connection header received
0928 11:02:40.316762 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.316766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.316841 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.316924 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:40.317508 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.318260 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.319716 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.320087 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.320176 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.321241 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:40.321246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:40.321250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:40.321267 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:40.321300 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:40.321312 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:40.321440 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.321701 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.321751 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 11:02:40.326267 22884 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:02:40.333101 22884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:02:40.366622 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.345655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:32775 (local address 127.0.0.1:38526)
0928 11:02:40.358971 (+ 13316us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.358991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:40.359106 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:40.359663 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:40.359668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:40.359690 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:40.360018 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.360028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.362030 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.362035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.363716 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.363729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.365586 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.365592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.365783 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.365860 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 11:02:40.365969 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 11:02:40.366175 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13209,"mutex_wait_us":2,"thread_start_us":11064,"threads_started":1}
I20260928 11:02:40.366806 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.345203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.3.252.65:39451)
0928 11:02:40.351995 (+ 6792us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.352005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.357319 (+ 5314us) server_negotiation.cc:407] Connection header received
0928 11:02:40.358340 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.358349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.358618 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.358795 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:40.359715 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.361152 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.363963 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.365372 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.365504 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.365775 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.366289 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.366466 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6594,"thread_start_us":604,"threads_started":1}
I20260928 11:02:40.367172 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.345587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40251 (local address 127.0.0.1:56952)
0928 11:02:40.358036 (+ 12449us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.358060 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:40.358191 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:40.360191 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:40.360195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:40.360220 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:40.360528 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.360539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.362632 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.362638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.364461 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.364471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.366383 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.366390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.366580 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.366620 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 11:02:40.366709 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 11:02:40.366829 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12289,"mutex_wait_us":3,"thread_start_us":12710,"threads_started":1}
I20260928 11:02:40.367311 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.346275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.3.252.67:40251)
0928 11:02:40.359653 (+ 13378us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.359661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.359691 (+ 30us) server_negotiation.cc:407] Connection header received
0928 11:02:40.359780 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.359787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.359993 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.360180 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:40.360741 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.362481 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.364630 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.365927 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.366049 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.366257 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.366794 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.366952 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13205,"thread_start_us":13052,"threads_started":1}
I20260928 11:02:40.366657 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.345149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54308)
0928 11:02:40.357137 (+ 11988us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.357184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:40.358062 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:40.359091 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:40.359098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:40.359123 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:40.359518 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.359532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.361306 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.361311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.363733 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.363749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.365917 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.365923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.366096 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.366126 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 11:02:40.366206 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 11:02:40.366317 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11710,"mutex_wait_us":61,"thread_start_us":13574,"threads_started":1}
I20260928 11:02:40.368265 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.345976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.3.252.66:32775)
0928 11:02:40.357729 (+ 11753us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.357738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.359126 (+ 1388us) server_negotiation.cc:407] Connection header received
0928 11:02:40.359250 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.359258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.359465 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.359663 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:40.360169 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.361878 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.363909 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.365372 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.367696 (+ 2324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.367856 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.367913 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.368023 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11582,"thread_start_us":11751,"threads_started":1}
I20260928 11:02:40.371016 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 7c96af260e3e4e8d9d039d1ac959ee37 (DEFAULT_TABLE table=test-table [id=ef7071f8b336443a8196b18bda7b8492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:40.371091 23005 tablet_service.cc:1467] Processing CreateTablet for tablet 7c96af260e3e4e8d9d039d1ac959ee37 (DEFAULT_TABLE table=test-table [id=ef7071f8b336443a8196b18bda7b8492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:40.371410 23136 tablet_service.cc:1467] Processing CreateTablet for tablet 7c96af260e3e4e8d9d039d1ac959ee37 (DEFAULT_TABLE table=test-table [id=ef7071f8b336443a8196b18bda7b8492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:40.372411 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c96af260e3e4e8d9d039d1ac959ee37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:40.372412 23005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c96af260e3e4e8d9d039d1ac959ee37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:40.372453 23136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c96af260e3e4e8d9d039d1ac959ee37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:40.384593 23349 tablet_bootstrap.cc:492] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Bootstrap starting.
I20260928 11:02:40.388598 23349 tablet_bootstrap.cc:654] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:40.391678 23349 log.cc:824] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:40.394764 23350 tablet_bootstrap.cc:492] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: Bootstrap starting.
I20260928 11:02:40.396364 23351 tablet_bootstrap.cc:492] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Bootstrap starting.
I20260928 11:02:40.398131 23349 tablet_bootstrap.cc:492] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: No bootstrap required, opened a new log
I20260928 11:02:40.398420 23349 ts_tablet_manager.cc:1397] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260928 11:02:40.400033 23351 tablet_bootstrap.cc:654] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:40.400501 23350 tablet_bootstrap.cc:654] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:40.401280 23351 log.cc:824] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:40.401388 23350 log.cc:824] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:40.404273 23350 tablet_bootstrap.cc:492] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: No bootstrap required, opened a new log
I20260928 11:02:40.404273 23351 tablet_bootstrap.cc:492] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: No bootstrap required, opened a new log
I20260928 11:02:40.404544 23351 ts_tablet_manager.cc:1397] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 11:02:40.404546 23350 ts_tablet_manager.cc:1397] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260928 11:02:40.407295 23349 raft_consensus.cc:348] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.407718 23349 raft_consensus.cc:374] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:40.407824 23349 raft_consensus.cc:729] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4567f1c1347e4f37a04d403cb1435284, State: Initialized, Role: FOLLOWER
I20260928 11:02:40.408392 23349 consensus_queue.cc:260] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.409416 23331 heartbeater.cc:503] Master 127.3.252.126:33393 was elected leader, sending a full tablet report...
I20260928 11:02:40.409468 23070 heartbeater.cc:503] Master 127.3.252.126:33393 was elected leader, sending a full tablet report...
I20260928 11:02:40.410123 23349 ts_tablet_manager.cc:1428] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260928 11:02:40.414348 23351 raft_consensus.cc:348] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.414783 23351 raft_consensus.cc:374] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:40.414891 23351 raft_consensus.cc:729] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab00aad45259413c8c9cc7fb8e2eabfc, State: Initialized, Role: FOLLOWER
I20260928 11:02:40.415515 23351 consensus_queue.cc:260] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.416837 23201 heartbeater.cc:503] Master 127.3.252.126:33393 was elected leader, sending a full tablet report...
I20260928 11:02:40.417200 23351 ts_tablet_manager.cc:1428] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 11:02:40.418273 23350 raft_consensus.cc:348] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.418704 23350 raft_consensus.cc:374] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:40.418817 23350 raft_consensus.cc:729] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e1ed0a75364241a27a558b2fb03586, State: Initialized, Role: FOLLOWER
I20260928 11:02:40.419404 23350 consensus_queue.cc:260] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.421097 23350 ts_tablet_manager.cc:1428] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260928 11:02:40.475019 23356 raft_consensus.cc:482] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:40.475363 23356 raft_consensus.cc:504] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.477053 23356 leader_election.cc:288] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61e1ed0a75364241a27a558b2fb03586 (127.3.252.65:39451), 4567f1c1347e4f37a04d403cb1435284 (127.3.252.67:40251)
I20260928 11:02:40.485143 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.477558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:35117 (local address 127.3.252.67:40251)
0928 11:02:40.477774 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.477780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.478431 (+ 651us) server_negotiation.cc:407] Connection header received
0928 11:02:40.478640 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.478646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.478713 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.478797 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:40.479340 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.480335 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.482923 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.484216 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.484643 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.484718 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.484775 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.484896 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 11:02:40.485194 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.477572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:56597 (local address 127.3.252.65:39451)
0928 11:02:40.477762 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.477767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.478783 (+ 1016us) server_negotiation.cc:407] Connection header received
0928 11:02:40.478930 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.478934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.479005 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.479073 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:40.479690 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.480625 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.483000 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.484200 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.484360 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.484429 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.484828 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.484958 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 11:02:40.485265 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.477550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:40251 (local address 127.3.252.66:35117)
0928 11:02:40.478317 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.478333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:40.478498 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:40.478830 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:40.478835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:40.478857 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:40.479194 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.479206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.480468 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.480472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.482716 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.482733 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.484397 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.484405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.484594 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.484670 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 11:02:40.484754 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 11:02:40.484888 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":30,"thread_start_us":416,"threads_started":1}
I20260928 11:02:40.485766 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.477550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.3.252.66:56597)
0928 11:02:40.478676 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.478690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:40.478805 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:40.479128 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:40.479133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:40.479161 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:40.479510 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.479519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.480754 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.480759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.482729 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:40.482739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.484454 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.484460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:40.484640 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.484691 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 11:02:40.485377 (+ 686us) client_negotiation.cc:240] Negotiation successful
0928 11:02:40.485504 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":283,"thread_start_us":536,"threads_started":1}
I20260928 11:02:40.486332 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c96af260e3e4e8d9d039d1ac959ee37" candidate_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4567f1c1347e4f37a04d403cb1435284" is_pre_election: true
I20260928 11:02:40.486837 23025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c96af260e3e4e8d9d039d1ac959ee37" candidate_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e1ed0a75364241a27a558b2fb03586" is_pre_election: true
I20260928 11:02:40.486949 23286 raft_consensus.cc:2454] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab00aad45259413c8c9cc7fb8e2eabfc in term 0.
I20260928 11:02:40.487357 23025 raft_consensus.cc:2454] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab00aad45259413c8c9cc7fb8e2eabfc in term 0.
I20260928 11:02:40.487674 23091 leader_election.cc:302] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4567f1c1347e4f37a04d403cb1435284, ab00aad45259413c8c9cc7fb8e2eabfc; no voters:
I20260928 11:02:40.488112 23356 raft_consensus.cc:2781] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:02:40.488260 23356 raft_consensus.cc:482] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:02:40.488351 23356 raft_consensus.cc:3037] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:40.490618 23356 raft_consensus.cc:504] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.491190 23356 leader_election.cc:288] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [CANDIDATE]: Term 1 election: Requested vote from peers 61e1ed0a75364241a27a558b2fb03586 (127.3.252.65:39451), 4567f1c1347e4f37a04d403cb1435284 (127.3.252.67:40251)
I20260928 11:02:40.491755 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c96af260e3e4e8d9d039d1ac959ee37" candidate_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4567f1c1347e4f37a04d403cb1435284"
I20260928 11:02:40.491755 23025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c96af260e3e4e8d9d039d1ac959ee37" candidate_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e1ed0a75364241a27a558b2fb03586"
I20260928 11:02:40.491968 23025 raft_consensus.cc:3037] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:40.492019 23286 raft_consensus.cc:3037] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:40.495164 23286 raft_consensus.cc:2454] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab00aad45259413c8c9cc7fb8e2eabfc in term 1.
I20260928 11:02:40.495165 23025 raft_consensus.cc:2454] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab00aad45259413c8c9cc7fb8e2eabfc in term 1.
I20260928 11:02:40.495615 23091 leader_election.cc:302] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4567f1c1347e4f37a04d403cb1435284, ab00aad45259413c8c9cc7fb8e2eabfc; no voters:
I20260928 11:02:40.495891 23356 raft_consensus.cc:2781] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:40.496369 23356 raft_consensus.cc:686] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 1 LEADER]: Becoming Leader. State: Replica: ab00aad45259413c8c9cc7fb8e2eabfc, State: Running, Role: LEADER
I20260928 11:02:40.496812 23356 consensus_queue.cc:237] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } }
I20260928 11:02:40.501647 22884 catalog_manager.cc:5495] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc reported cstate change: term changed from 0 to 1, leader changed from <none> to ab00aad45259413c8c9cc7fb8e2eabfc (127.3.252.66). New cstate: current_term: 1 leader_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab00aad45259413c8c9cc7fb8e2eabfc" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 32775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 } health_report { overall_health: UNKNOWN } } }
I20260928 11:02:40.527360 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.522322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.3.252.66:32775)
0928 11:02:40.522502 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.522507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.522594 (+ 87us) server_negotiation.cc:407] Connection header received
0928 11:02:40.522757 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.522761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.522851 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.522982 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:02:40.523313 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.523990 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.525350 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.525755 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.525870 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.526360 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:02:40.526867 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:02:40.527007 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.527113 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.527153 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 11:02:40.535372 23286 raft_consensus.cc:1261] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 1 FOLLOWER]: Refusing update from remote peer ab00aad45259413c8c9cc7fb8e2eabfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:40.535373 23025 raft_consensus.cc:1261] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 1 FOLLOWER]: Refusing update from remote peer ab00aad45259413c8c9cc7fb8e2eabfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:40.536310 23360 consensus_queue.cc:1035] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e1ed0a75364241a27a558b2fb03586" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:40.536619 23356 consensus_queue.cc:1035] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4567f1c1347e4f37a04d403cb1435284" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 40251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 11:02:40.539331 23332 tablet.cc:2369] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:02:40.548198 23363 mvcc.cc:205] Tried to move back new op lower bound from 7334270404742590464 to 7334270404598816768. Current Snapshot: MvccSnapshot[applied={T|T < 7334270404742590464}]
I20260928 11:02:40.548856 23364 mvcc.cc:205] Tried to move back new op lower bound from 7334270404742590464 to 7334270404598816768. Current Snapshot: MvccSnapshot[applied={T|T < 7334270404742590464}]
I20260928 11:02:40.562681 22884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:40.563235 22884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:40.567539 22884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8498e7c875f4c8dbef7bfb9d9532796: Sending DeleteTablet for 3 replicas of tablet 7c96af260e3e4e8d9d039d1ac959ee37
I20260928 11:02:40.568661 23005 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c96af260e3e4e8d9d039d1ac959ee37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:40 UTC) from {username='slave'} at 127.0.0.1:54308
I20260928 11:02:40.568702 23136 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c96af260e3e4e8d9d039d1ac959ee37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:40 UTC) from {username='slave'} at 127.0.0.1:38526
I20260928 11:02:40.568872 23266 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c96af260e3e4e8d9d039d1ac959ee37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:40 UTC) from {username='slave'} at 127.0.0.1:56952
I20260928 11:02:40.569571 23373 tablet_replica.cc:330] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: stopping tablet replica
I20260928 11:02:40.570084 23373 raft_consensus.cc:2229] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:40.570623 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 22943
I20260928 11:02:40.570685 23373 raft_consensus.cc:2258] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:40.578649 23375 tablet_replica.cc:330] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: stopping tablet replica
I20260928 11:02:40.579123 23375 raft_consensus.cc:2229] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:40.579679 23375 raft_consensus.cc:2258] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:40.579991 23373 ts_tablet_manager.cc:1905] T 7c96af260e3e4e8d9d039d1ac959ee37 P 61e1ed0a75364241a27a558b2fb03586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:40.580415 23374 tablet_replica.cc:330] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: stopping tablet replica
I20260928 11:02:40.580811 23374 raft_consensus.cc:2229] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:02:40.581418 23374 raft_consensus.cc:2258] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:40.583578 23375 ts_tablet_manager.cc:1905] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:40.583585 23374 ts_tablet_manager.cc:1905] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:40.586087 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.571345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.3.252.65:39451)
0928 11:02:40.571512 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.571519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.578338 (+ 6819us) server_negotiation.cc:407] Connection header received
0928 11:02:40.579158 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.579165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.579286 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.579383 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:40.579794 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.580684 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.582173 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.582788 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.582885 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.584330 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:40.584341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:40.584357 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:40.585101 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:40.585161 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:40.585186 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:40.585264 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.585503 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.585636 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 11:02:40.586947 23065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:40.590517 23375 ts_tablet_manager.cc:1918] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:40.590744 23375 log.cc:1196] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/7c96af260e3e4e8d9d039d1ac959ee37
I20260928 11:02:40.591085 23374 ts_tablet_manager.cc:1918] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:40.591356 23374 log.cc:1196] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/7c96af260e3e4e8d9d039d1ac959ee37
I20260928 11:02:40.591383 23375 ts_tablet_manager.cc:1939] T 7c96af260e3e4e8d9d039d1ac959ee37 P 4567f1c1347e4f37a04d403cb1435284: Deleting consensus metadata
I20260928 11:02:40.591928 23374 ts_tablet_manager.cc:1939] T 7c96af260e3e4e8d9d039d1ac959ee37 P ab00aad45259413c8c9cc7fb8e2eabfc: Deleting consensus metadata
I20260928 11:02:40.593417 22870 catalog_manager.cc:4841] TS 4567f1c1347e4f37a04d403cb1435284 (127.3.252.67:40251): tablet 7c96af260e3e4e8d9d039d1ac959ee37 (table test-table [id=ef7071f8b336443a8196b18bda7b8492]) successfully deleted
I20260928 11:02:40.593684 22869 catalog_manager.cc:4841] TS ab00aad45259413c8c9cc7fb8e2eabfc (127.3.252.66:32775): tablet 7c96af260e3e4e8d9d039d1ac959ee37 (table test-table [id=ef7071f8b336443a8196b18bda7b8492]) successfully deleted
I20260928 11:02:40.710014 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 22943
W20260928 11:02:40.724849 22868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.252.65:39451 (error 108)
I20260928 11:02:40.724954 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 23074
I20260928 11:02:40.726838 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.726391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54338)
0928 11:02:40.726545 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.726661 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 11:02:40.727185 22868 catalog_manager.cc:4584] TS 61e1ed0a75364241a27a558b2fb03586 (127.3.252.65:39451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c96af260e3e4e8d9d039d1ac959ee37: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
I20260928 11:02:40.733263 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.726327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.3.252.66:32775)
0928 11:02:40.726491 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.726496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.726515 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:02:40.726586 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.726591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.726692 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.726804 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:40.727325 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.728124 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.730021 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.730518 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.730630 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.731748 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:40.731756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:40.731771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:40.732399 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:40.732447 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:40.732468 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:40.732557 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.732779 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.732891 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 11:02:40.733924 23196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:40.781015 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.780547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54350)
0928 11:02:40.780763 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.780850 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 11:02:40.782114 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.781723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54366)
0928 11:02:40.781898 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.781979 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 11:02:40.842923 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.842375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54376)
0928 11:02:40.842588 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.842694 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 11:02:40.844303 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.843901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54378)
0928 11:02:40.844055 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.844158 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 11:02:40.844465 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 23074
I20260928 11:02:40.859426 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 23204
I20260928 11:02:40.866910 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.860248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.3.252.67:40251)
0928 11:02:40.860419 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:40.860423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:40.860478 (+ 55us) server_negotiation.cc:407] Connection header received
0928 11:02:40.860654 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:40.860659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:40.860743 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:40.860820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:40.861208 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.861992 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.863876 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:40.864295 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:40.864402 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:40.865490 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:40.865499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:40.865514 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:40.866212 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:40.866265 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:40.866286 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:40.866369 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:40.866511 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 11:02:40.866624 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 11:02:40.867614 23326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:40.925781 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.925310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54394)
0928 11:02:40.925540 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.925630 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 11:02:40.926831 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:40.926430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:39451 (local address 127.0.0.1:54398)
0928 11:02:40.926625 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:40.926721 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 11:02:40.955654 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 23204
I20260928 11:02:40.969638 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 22853
I20260928 11:02:40.970757 22914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:41.079022 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 22853
2026-09-28T11:02:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2699 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:41 dist-test-slave-djjw krb5kdc[23384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:41 dist-test-slave-djjw krb5kdc[23384](info): set up 2 sockets
Sep 28 11:02:41 dist-test-slave-djjw krb5kdc[23384](info): commencing operation
krb5kdc: starting...
W20260928 11:02:42.098109 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:42 dist-test-slave-djjw krb5kdc[23384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:02:42.345577 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36953
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:42.666235 23397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:42.666518 23397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:42.666661 23397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:42.673817 23397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:42.673943 23397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:42.674013 23397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:42.674088 23397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:42.674155 23397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:42.683980 23397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:42.685297 23397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:42.686610 23397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:42.698144 23403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:42.698216 23405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:42.698216 23402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:42.699054 23397 server_base.cc:1034] running on GCE node
I20260928 11:02:42.699819 23397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:42.701220 23397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:42.702412 23397 hybrid_clock.cc:648] HybridClock initialized: now 1790593362702361 us; error 35 us; skew 500 ppm
I20260928 11:02:42.702737 23397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:42 dist-test-slave-djjw krb5kdc[23384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593362, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:42.708803 23397 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:02:42.710276 23397 webserver.cc:460] Webserver started at http://127.3.252.126:43337/ using document root <none> and password file <none>
I20260928 11:02:42.710973 23397 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:42.711086 23397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:42.711372 23397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:42.714470 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "432a362a7bdf43debb3c90fae21a4c2f"
format_stamp: "Formatted at 2026-09-28 11:02:42 on dist-test-slave-djjw"
server_key: "d8b55f908fe08660ab886db8388d0854"
server_key_iv: "35c9a1838c8aad7d5337040a96759d88"
server_key_version: "encryptionkey@0"
I20260928 11:02:42.715184 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "432a362a7bdf43debb3c90fae21a4c2f"
format_stamp: "Formatted at 2026-09-28 11:02:42 on dist-test-slave-djjw"
server_key: "d8b55f908fe08660ab886db8388d0854"
server_key_iv: "35c9a1838c8aad7d5337040a96759d88"
server_key_version: "encryptionkey@0"
I20260928 11:02:42.720291 23397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:02:42.723774 23412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:42.725083 23397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:02:42.725240 23397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "432a362a7bdf43debb3c90fae21a4c2f"
format_stamp: "Formatted at 2026-09-28 11:02:42 on dist-test-slave-djjw"
server_key: "d8b55f908fe08660ab886db8388d0854"
server_key_iv: "35c9a1838c8aad7d5337040a96759d88"
server_key_version: "encryptionkey@0"
I20260928 11:02:42.725419 23397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:42.741331 23397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:42.745602 23397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:42.745903 23397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:42.762914 23397 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:36953
I20260928 11:02:42.762944 23463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:36953 every 8 connection(s)
I20260928 11:02:42.764196 23397 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:42.767882 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 23397
I20260928 11:02:42.767944 23464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:42.768044 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:42.768482 4081 external_mini_cluster.cc:1428] Setting key f29f75baa5caac4a81a2479212a7227e
I20260928 11:02:42.782130 23464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f: Bootstrap starting.
Sep 28 11:02:42 dist-test-slave-djjw krb5kdc[23384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:42.785637 23464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:42.786737 23464 log.cc:824] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:42.789602 23464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f: No bootstrap required, opened a new log
I20260928 11:02:42.791265 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:42.770232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.3.252.126:36953)
0928 11:02:42.771065 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:42.771076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:42.771122 (+ 46us) server_negotiation.cc:407] Connection header received
0928 11:02:42.772785 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:42.772809 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:42.773218 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:42.773565 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:42.775439 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:42.776693 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:42.778466 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:42.779160 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:42.779316 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:42.784140 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:42.784149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:42.784164 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:42.784204 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:42.787251 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:42.787978 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:42.787986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:42.787992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:42.788169 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:42.788550 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:42.788553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:42.788555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:42.789094 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:42.789376 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:42.789803 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 11:02:42.790273 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":363,"threads_started":1}
I20260928 11:02:42.797415 23464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432a362a7bdf43debb3c90fae21a4c2f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36953 } }
I20260928 11:02:42.797814 23464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:42.797951 23464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432a362a7bdf43debb3c90fae21a4c2f, State: Initialized, Role: FOLLOWER
I20260928 11:02:42.798501 23464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432a362a7bdf43debb3c90fae21a4c2f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36953 } }
I20260928 11:02:42.798727 23464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:42.798849 23464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:42.798997 23464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:42.801594 23464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432a362a7bdf43debb3c90fae21a4c2f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36953 } }
I20260928 11:02:42.802098 23464 leader_election.cc:302] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 432a362a7bdf43debb3c90fae21a4c2f; no voters:
I20260928 11:02:42.802649 23464 leader_election.cc:288] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:42.802765 23469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:42.803400 23469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [term 1 LEADER]: Becoming Leader. State: Replica: 432a362a7bdf43debb3c90fae21a4c2f, State: Running, Role: LEADER
I20260928 11:02:42.803900 23469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432a362a7bdf43debb3c90fae21a4c2f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36953 } }
I20260928 11:02:42.804647 23464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:42.806849 23471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 432a362a7bdf43debb3c90fae21a4c2f. Latest consensus state: current_term: 1 leader_uuid: "432a362a7bdf43debb3c90fae21a4c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432a362a7bdf43debb3c90fae21a4c2f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36953 } } }
I20260928 11:02:42.806924 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "432a362a7bdf43debb3c90fae21a4c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432a362a7bdf43debb3c90fae21a4c2f" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36953 } } }
I20260928 11:02:42.808420 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:42.807174 23471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:42.809367 23479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:42.814313 23479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:42.822678 23479 catalog_manager.cc:1347] Generated new cluster ID: 59291d71348d4fbc899f42d7d0afff91
I20260928 11:02:42.822817 23479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:42.843636 23479 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:42.845563 23479 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:42.853585 23479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f: Generated new TSK 0
I20260928 11:02:42.854486 23479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.252.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
Entry for principal kudu/127.3.252.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab.
I20260928 11:02:42.920300 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36953
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:43.284756 23490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:43.285051 23490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:43.285144 23490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:43.292764 23490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:43.292925 23490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:43.293082 23490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:02:43.303279 23490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.65.keytab
--principal=kudu/127.3.252.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:43.304560 23490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:43.305924 23490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:43.319168 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:43.319281 23496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:43.319245 23495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:43.320320 23490 server_base.cc:1034] running on GCE node
I20260928 11:02:43.320922 23490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:43.321951 23490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:43.323149 23490 hybrid_clock.cc:648] HybridClock initialized: now 1790593363323099 us; error 48 us; skew 500 ppm
I20260928 11:02:43.323465 23490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:43 dist-test-slave-djjw krb5kdc[23384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:43.326562 23490 init.cc:377] Logged in from keytab as kudu/127.3.252.65@KRBTEST.COM (short username kudu)
I20260928 11:02:43.328055 23490 webserver.cc:460] Webserver started at http://127.3.252.65:46349/ using document root <none> and password file <none>
I20260928 11:02:43.328723 23490 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:43.328838 23490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:43.329164 23490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:43.332315 23490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "6744afeda3b14eeb80cd4e09a5ac7a50"
format_stamp: "Formatted at 2026-09-28 11:02:43 on dist-test-slave-djjw"
server_key: "76abdd0db8918680cca4d043fa6cb67b"
server_key_iv: "27b98e4c0f1070a1cd24028263f58e9d"
server_key_version: "encryptionkey@0"
I20260928 11:02:43.333108 23490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "6744afeda3b14eeb80cd4e09a5ac7a50"
format_stamp: "Formatted at 2026-09-28 11:02:43 on dist-test-slave-djjw"
server_key: "76abdd0db8918680cca4d043fa6cb67b"
server_key_iv: "27b98e4c0f1070a1cd24028263f58e9d"
server_key_version: "encryptionkey@0"
I20260928 11:02:43.338459 23490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 11:02:43.341936 23505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:43.343365 23490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:43.343524 23490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "6744afeda3b14eeb80cd4e09a5ac7a50"
format_stamp: "Formatted at 2026-09-28 11:02:43 on dist-test-slave-djjw"
server_key: "76abdd0db8918680cca4d043fa6cb67b"
server_key_iv: "27b98e4c0f1070a1cd24028263f58e9d"
server_key_version: "encryptionkey@0"
I20260928 11:02:43.343714 23490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:43.373634 23490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:43.377926 23490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:43.378192 23490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:43.379150 23490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:43.380443 23490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:43.380522 23490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:43.380635 23490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:43.380693 23490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:43.410899 23490 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:41967
I20260928 11:02:43.410943 23617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:41967 every 8 connection(s)
I20260928 11:02:43.412143 23490 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:02:43.419375 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 23490
I20260928 11:02:43.419535 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:02:43.419951 4081 external_mini_cluster.cc:1428] Setting key 5c81f72792bbacaae68efa69d0469c51
Sep 28 11:02:43 dist-test-slave-djjw krb5kdc[23384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.65@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:43.432555 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:43.414649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:39209 (local address 127.3.252.126:36953)
0928 11:02:43.415053 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:43.415057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:43.415995 (+ 938us) server_negotiation.cc:407] Connection header received
0928 11:02:43.417914 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:43.417919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:43.418008 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:43.418106 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:43.419631 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.420429 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.422457 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.422854 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.422953 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:43.426843 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:43.426848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:43.426851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:43.426886 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:43.429448 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:43.430264 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:43.430268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:43.430270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:43.430390 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:43.430820 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:43.430824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:43.430826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:43.431097 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:43.431226 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:43.431958 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 11:02:43.432189 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":197,"threads_started":1}
I20260928 11:02:43.433597 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:43.415178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36953 (local address 127.3.252.65:39209)
0928 11:02:43.415849 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:43.415890 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:43.417609 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:43.418354 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:43.418363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:43.418880 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:43.419446 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:43.419469 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.420630 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.420635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:43.422314 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:43.422329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.423082 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.423089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:43.423385 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:43.424091 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:43.424121 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:43.426595 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:43.429602 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:43.429611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:43.429631 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:43.430146 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:43.430493 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:43.430498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:43.430502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:43.430695 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:43.431220 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:43.431228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:43.431584 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 11:02:43.431823 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 11:02:43.432400 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":255,"threads_started":1}
I20260928 11:02:43.435225 23618 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36953
I20260928 11:02:43.435559 23618 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:43.436440 23618 heartbeater.cc:511] Master 127.3.252.126:36953 requested a full tablet report, sending...
I20260928 11:02:43.438985 23429 ts_manager.cc:194] Registered new tserver with Master: 6744afeda3b14eeb80cd4e09a5ac7a50 (127.3.252.65:41967)
I20260928 11:02:43.440968 23429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.65@KRBTEST.COM'} at 127.3.252.65:39209
WARNING: no policy specified for kudu/127.3.252.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
Entry for principal kudu/127.3.252.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab.
I20260928 11:02:43.474186 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36953
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:43.796270 23624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:43.796586 23624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:43.796703 23624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:43.803929 23624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:43.804039 23624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:43.804163 23624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:02:43.813889 23624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.66.keytab
--principal=kudu/127.3.252.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:43.815096 23624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:43.816357 23624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:43.829339 23630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:43.829344 23629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:43.830015 23624 server_base.cc:1034] running on GCE node
W20260928 11:02:43.830147 23632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:43.830797 23624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:43.831733 23624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:43.832932 23624 hybrid_clock.cc:648] HybridClock initialized: now 1790593363832880 us; error 42 us; skew 500 ppm
I20260928 11:02:43.833225 23624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:43 dist-test-slave-djjw krb5kdc[23384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:43.836223 23624 init.cc:377] Logged in from keytab as kudu/127.3.252.66@KRBTEST.COM (short username kudu)
I20260928 11:02:43.837641 23624 webserver.cc:460] Webserver started at http://127.3.252.66:43887/ using document root <none> and password file <none>
I20260928 11:02:43.838354 23624 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:43.838459 23624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:43.838749 23624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:43.841732 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "769051d203a345e4ae36556b7a01ea8a"
format_stamp: "Formatted at 2026-09-28 11:02:43 on dist-test-slave-djjw"
server_key: "ed6d807e292114788729731303fd7271"
server_key_iv: "1be9192ba1be9bb3c7be7472e9728a5a"
server_key_version: "encryptionkey@0"
I20260928 11:02:43.842514 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "769051d203a345e4ae36556b7a01ea8a"
format_stamp: "Formatted at 2026-09-28 11:02:43 on dist-test-slave-djjw"
server_key: "ed6d807e292114788729731303fd7271"
server_key_iv: "1be9192ba1be9bb3c7be7472e9728a5a"
server_key_version: "encryptionkey@0"
I20260928 11:02:43.847798 23624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:02:43.851277 23639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:43.852509 23624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 11:02:43.852666 23624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "769051d203a345e4ae36556b7a01ea8a"
format_stamp: "Formatted at 2026-09-28 11:02:43 on dist-test-slave-djjw"
server_key: "ed6d807e292114788729731303fd7271"
server_key_iv: "1be9192ba1be9bb3c7be7472e9728a5a"
server_key_version: "encryptionkey@0"
I20260928 11:02:43.852849 23624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:43.872329 23624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:43.876267 23624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:43.876508 23624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:43.877417 23624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:43.878783 23624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:43.878855 23624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:43.878948 23624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:43.878988 23624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:43.908545 23624 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:41369
I20260928 11:02:43.908614 23751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:41369 every 8 connection(s)
I20260928 11:02:43.909857 23624 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:02:43.918885 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 23624
I20260928 11:02:43.919081 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:02:43.919600 4081 external_mini_cluster.cc:1428] Setting key c747aa54030b3e52ad03593929d7585b
Sep 28 11:02:43 dist-test-slave-djjw krb5kdc[23384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.66@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:43.929994 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:43.912324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:34325 (local address 127.3.252.126:36953)
0928 11:02:43.912522 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:43.912527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:43.913670 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 11:02:43.915627 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:43.915633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:43.915719 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:43.915806 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:43.917272 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.918024 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.920041 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.920599 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.920719 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:43.924562 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:43.924567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:43.924570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:43.924605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:43.926993 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:43.927699 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:43.927705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:43.927707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:43.927831 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:43.928223 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:43.928228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:43.928231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:43.928571 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:43.928700 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:43.929390 (+ 690us) server_negotiation.cc:299] Negotiation successful
0928 11:02:43.929624 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 11:02:43.930994 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:43.912807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36953 (local address 127.3.252.66:34325)
0928 11:02:43.913493 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:43.913536 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:43.915361 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:43.916047 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:43.916058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:43.916554 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:43.917102 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:43.917120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.918176 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.918180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:43.919899 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:43.919915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:43.920810 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:43.920815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:43.921081 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:43.921857 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:43.921895 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:43.924335 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:43.927132 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:43.927142 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:43.927160 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:43.927582 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:43.927931 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:43.927936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:43.927940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:43.928120 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:43.928686 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:43.928694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:43.929041 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 11:02:43.929280 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 11:02:43.929900 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260928 11:02:43.932300 23752 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36953
I20260928 11:02:43.932611 23752 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:43.933403 23752 heartbeater.cc:511] Master 127.3.252.126:36953 requested a full tablet report, sending...
I20260928 11:02:43.934898 23429 ts_manager.cc:194] Registered new tserver with Master: 769051d203a345e4ae36556b7a01ea8a (127.3.252.66:41369)
I20260928 11:02:43.935674 23429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.66@KRBTEST.COM'} at 127.3.252.66:34325
WARNING: no policy specified for kudu/127.3.252.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
Entry for principal kudu/127.3.252.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab.
I20260928 11:02:43.973338 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36953
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:44.296905 23757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:44.297220 23757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:44.297354 23757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:44.304919 23757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:44.305055 23757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:44.305214 23757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:02:44.315752 23757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.67.keytab
--principal=kudu/127.3.252.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:36953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:44.317049 23757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:44.318354 23757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:44.331308 23765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:44.331359 23762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:44.331359 23763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:44.332186 23757 server_base.cc:1034] running on GCE node
I20260928 11:02:44.332777 23757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:44.333693 23757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:44.334915 23757 hybrid_clock.cc:648] HybridClock initialized: now 1790593364334860 us; error 62 us; skew 500 ppm
I20260928 11:02:44.335238 23757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:44 dist-test-slave-djjw krb5kdc[23384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:44.338274 23757 init.cc:377] Logged in from keytab as kudu/127.3.252.67@KRBTEST.COM (short username kudu)
I20260928 11:02:44.339743 23757 webserver.cc:460] Webserver started at http://127.3.252.67:40715/ using document root <none> and password file <none>
I20260928 11:02:44.340395 23757 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:44.340490 23757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:44.340742 23757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:44.343715 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "e86d32078e1d4ace92f3e89c96071435"
format_stamp: "Formatted at 2026-09-28 11:02:44 on dist-test-slave-djjw"
server_key: "16976a72ec2a81ecbc8325900e091d6f"
server_key_iv: "0030f07f90f3cc604230d19b48c2b47e"
server_key_version: "encryptionkey@0"
I20260928 11:02:44.344491 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "e86d32078e1d4ace92f3e89c96071435"
format_stamp: "Formatted at 2026-09-28 11:02:44 on dist-test-slave-djjw"
server_key: "16976a72ec2a81ecbc8325900e091d6f"
server_key_iv: "0030f07f90f3cc604230d19b48c2b47e"
server_key_version: "encryptionkey@0"
I20260928 11:02:44.349735 23757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 11:02:44.353298 23772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:44.354650 23757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:44.354810 23757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "e86d32078e1d4ace92f3e89c96071435"
format_stamp: "Formatted at 2026-09-28 11:02:44 on dist-test-slave-djjw"
server_key: "16976a72ec2a81ecbc8325900e091d6f"
server_key_iv: "0030f07f90f3cc604230d19b48c2b47e"
server_key_version: "encryptionkey@0"
I20260928 11:02:44.354990 23757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:44.388464 23757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:44.392757 23757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:44.393005 23757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:44.393987 23757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:44.395283 23757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:44.395357 23757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:44.395457 23757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:44.395499 23757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:44.425760 23757 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:43983
I20260928 11:02:44.425801 23884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:43983 every 8 connection(s)
I20260928 11:02:44.427106 23757 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:02:44.434623 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 23757
I20260928 11:02:44.434824 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:02:44.435253 4081 external_mini_cluster.cc:1428] Setting key 3cbd4058c600abc696a90fba24233745
Sep 28 11:02:44 dist-test-slave-djjw krb5kdc[23384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.67@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:02:44.444810 23618 heartbeater.cc:503] Master 127.3.252.126:36953 was elected leader, sending a full tablet report...
I20260928 11:02:44.447470 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.429720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:49403 (local address 127.3.252.126:36953)
0928 11:02:44.429957 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.429962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.431237 (+ 1275us) server_negotiation.cc:407] Connection header received
0928 11:02:44.433188 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.433194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.433311 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.433402 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:44.435123 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.435938 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.438053 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.438449 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.438542 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.441877 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:44.441882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:44.441885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:44.441919 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:44.444345 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:44.445184 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:44.445189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:44.445192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:44.445348 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:44.445803 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:44.445807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:44.445810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:44.446172 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:44.446309 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.447051 (+ 742us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.447227 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 11:02:44.448451 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.430310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36953 (local address 127.3.252.67:49403)
0928 11:02:44.431047 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.431092 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.432902 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.433680 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.433688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.434310 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:44.434950 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.434970 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.436096 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.436102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.437918 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.437932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.438659 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.438665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.438867 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.439490 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:02:44.439516 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:44.441621 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:44.444471 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:44.444478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:44.444491 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:44.445059 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:44.445447 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:02:44.445451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:02:44.445454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:02:44.445682 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:02:44.446287 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:44.446294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:44.446721 (+ 427us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.446945 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.447348 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
I20260928 11:02:44.449702 23885 heartbeater.cc:347] Connected to a master server at 127.3.252.126:36953
I20260928 11:02:44.450050 23885 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:44.450799 23885 heartbeater.cc:511] Master 127.3.252.126:36953 requested a full tablet report, sending...
I20260928 11:02:44.452077 23428 ts_manager.cc:194] Registered new tserver with Master: e86d32078e1d4ace92f3e89c96071435 (127.3.252.67:43983)
I20260928 11:02:44.452875 23428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.252.67@KRBTEST.COM'} at 127.3.252.67:49403
I20260928 11:02:44.462620 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:02:44.478598 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.466560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.3.252.126:36953)
0928 11:02:44.466762 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.466768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.467070 (+ 302us) server_negotiation.cc:407] Connection header received
0928 11:02:44.467238 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.467243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.467319 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.467399 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:44.468731 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.469462 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.470769 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.471164 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.471275 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.473479 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:44.473483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:44.473486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:44.473518 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:44.476023 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:44.476644 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:44.476649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:44.476653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:44.476786 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:44.477134 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:44.477138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:44.477141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:44.477468 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:44.477697 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.478077 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.478300 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 11:02:44.482427 23428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:02:44.489279 23428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:02:44.513393 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.501876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:41967 (local address 127.0.0.1:35452)
0928 11:02:44.503562 (+ 1686us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.503578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.503775 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.504700 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.504705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.504732 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:44.505181 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.505192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.507377 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.507384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.510141 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.510160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.512398 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.512405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.512609 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.512664 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.512805 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.512994 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1546,"mutex_wait_us":43,"thread_start_us":1802,"threads_started":1}
I20260928 11:02:44.513396 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.502564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.3.252.65:41967)
0928 11:02:44.503326 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.503334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.503664 (+ 330us) server_negotiation.cc:407] Connection header received
0928 11:02:44.503967 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.503975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.504214 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.504389 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:44.505376 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.507172 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.510363 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.512072 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.512208 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.512469 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.512881 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.513065 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":458,"threads_started":1}
I20260928 11:02:44.514833 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.501869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.3.252.66:41369)
0928 11:02:44.503746 (+ 1877us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.503754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.503781 (+ 27us) server_negotiation.cc:407] Connection header received
0928 11:02:44.503884 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.503891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.504140 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.505298 (+ 1158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:44.505564 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.508581 (+ 3017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.512439 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.513848 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.513984 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.514182 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.514409 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.514561 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1706,"thread_start_us":1602,"threads_started":1}
I20260928 11:02:44.514870 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.501623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:41369 (local address 127.0.0.1:60886)
0928 11:02:44.502747 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.502800 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.502996 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.504472 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.504480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.504519 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:44.505043 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.505060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.508796 (+ 3736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.508804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.512241 (+ 3437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.512259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.514055 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.514061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.514229 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.514258 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.514341 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.514470 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":135,"thread_start_us":608,"threads_started":1}
I20260928 11:02:44.518208 23553 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2c04448fef4f52b7221b5ef09b9c13 (DEFAULT_TABLE table=test-table [id=c21253d924cf4f97ac04a4ed29c1f0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:44.519526 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2c04448fef4f52b7221b5ef09b9c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:44.519002 23687 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2c04448fef4f52b7221b5ef09b9c13 (DEFAULT_TABLE table=test-table [id=c21253d924cf4f97ac04a4ed29c1f0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:44.520349 23687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2c04448fef4f52b7221b5ef09b9c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:44.520495 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.504087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:43983 (local address 127.0.0.1:51382)
0928 11:02:44.512435 (+ 8348us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.512451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.512561 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.513479 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.513483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.513508 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:44.513820 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.513862 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.515772 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.515778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.517672 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.517688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.519484 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.519491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.519813 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.519847 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.519949 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.520084 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8168,"thread_start_us":8025,"threads_started":1}
I20260928 11:02:44.521169 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.504663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.3.252.67:43983)
0928 11:02:44.511946 (+ 7283us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.511955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.512549 (+ 594us) server_negotiation.cc:407] Connection header received
0928 11:02:44.512704 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.512712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.512995 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.513197 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:44.514079 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.515611 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.517876 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.519270 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.519404 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.519629 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.520638 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.520810 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6288,"thread_start_us":6125,"threads_started":1}
I20260928 11:02:44.524967 23820 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2c04448fef4f52b7221b5ef09b9c13 (DEFAULT_TABLE table=test-table [id=c21253d924cf4f97ac04a4ed29c1f0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:44.526698 23820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2c04448fef4f52b7221b5ef09b9c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:44.533092 23903 tablet_bootstrap.cc:492] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Bootstrap starting.
I20260928 11:02:44.537215 23903 tablet_bootstrap.cc:654] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:44.538136 23904 tablet_bootstrap.cc:492] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Bootstrap starting.
I20260928 11:02:44.538596 23903 log.cc:824] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:44.541857 23904 tablet_bootstrap.cc:654] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:44.541818 23906 tablet_bootstrap.cc:492] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Bootstrap starting.
I20260928 11:02:44.541960 23903 tablet_bootstrap.cc:492] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: No bootstrap required, opened a new log
I20260928 11:02:44.542654 23903 ts_tablet_manager.cc:1397] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 11:02:44.543370 23904 log.cc:824] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:44.544955 23906 tablet_bootstrap.cc:654] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:44.545989 23906 log.cc:824] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:44.549149 23904 tablet_bootstrap.cc:492] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: No bootstrap required, opened a new log
I20260928 11:02:44.549292 23906 tablet_bootstrap.cc:492] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: No bootstrap required, opened a new log
I20260928 11:02:44.549440 23904 ts_tablet_manager.cc:1397] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260928 11:02:44.549566 23906 ts_tablet_manager.cc:1397] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 11:02:44.554073 23903 raft_consensus.cc:348] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.554646 23903 raft_consensus.cc:374] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:44.554741 23903 raft_consensus.cc:729] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769051d203a345e4ae36556b7a01ea8a, State: Initialized, Role: FOLLOWER
I20260928 11:02:44.555487 23903 consensus_queue.cc:260] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.556659 23752 heartbeater.cc:503] Master 127.3.252.126:36953 was elected leader, sending a full tablet report...
I20260928 11:02:44.557487 23904 raft_consensus.cc:348] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.558035 23904 raft_consensus.cc:374] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:44.558142 23904 raft_consensus.cc:729] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6744afeda3b14eeb80cd4e09a5ac7a50, State: Initialized, Role: FOLLOWER
I20260928 11:02:44.558591 23903 ts_tablet_manager.cc:1428] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260928 11:02:44.558827 23904 consensus_queue.cc:260] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.559794 23906 raft_consensus.cc:348] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.560212 23906 raft_consensus.cc:374] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:44.560297 23906 raft_consensus.cc:729] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86d32078e1d4ace92f3e89c96071435, State: Initialized, Role: FOLLOWER
I20260928 11:02:44.560742 23904 ts_tablet_manager.cc:1428] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260928 11:02:44.560988 23906 consensus_queue.cc:260] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.562115 23885 heartbeater.cc:503] Master 127.3.252.126:36953 was elected leader, sending a full tablet report...
I20260928 11:02:44.562851 23906 ts_tablet_manager.cc:1428] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260928 11:02:44.661769 23753 tablet.cc:2369] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 11:02:44.664512 23619 tablet.cc:2369] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 11:02:44.678789 23886 tablet.cc:2369] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:02:44.683910 23911 raft_consensus.cc:482] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:44.684206 23911 raft_consensus.cc:504] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.685918 23911 leader_election.cc:288] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769051d203a345e4ae36556b7a01ea8a (127.3.252.66:41369), 6744afeda3b14eeb80cd4e09a5ac7a50 (127.3.252.65:41967)
I20260928 11:02:44.691407 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.686352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:38303 (local address 127.3.252.66:41369)
0928 11:02:44.686590 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.686597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.686810 (+ 213us) server_negotiation.cc:407] Connection header received
0928 11:02:44.686997 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.687001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.687060 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.687148 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:44.687524 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.688233 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.689795 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.690585 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.690727 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.690810 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.691046 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.691164 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 11:02:44.691437 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.686392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:41369 (local address 127.3.252.67:38303)
0928 11:02:44.686719 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.686734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.686863 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.687184 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.687188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.687215 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:44.687398 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.687406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.688355 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.688360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.689636 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.689651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.690726 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.690731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.690908 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.690933 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.691010 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.691140 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3}
I20260928 11:02:44.691921 23910 raft_consensus.cc:482] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:44.692447 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.686352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:48219 (local address 127.3.252.65:41967)
0928 11:02:44.686576 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.686583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.687346 (+ 763us) server_negotiation.cc:407] Connection header received
0928 11:02:44.687512 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.687517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.687579 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.687660 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:44.688148 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.688839 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.690519 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.691265 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.691404 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.691471 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.692035 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.692163 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:02:44.692230 23910 raft_consensus.cc:504] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.692453 23707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c04448fef4f52b7221b5ef09b9c13" candidate_uuid: "e86d32078e1d4ace92f3e89c96071435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769051d203a345e4ae36556b7a01ea8a" is_pre_election: true
I20260928 11:02:44.692755 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.686392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:41967 (local address 127.3.252.67:48219)
0928 11:02:44.687239 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.687254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.687387 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.687687 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.687690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.687713 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:44.688004 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.688012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.688975 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.688979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.690354 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.690366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.691448 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.691454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.691891 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.691915 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.692006 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.692211 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":86,"thread_start_us":416,"threads_started":1}
I20260928 11:02:44.693025 23707 raft_consensus.cc:2454] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e86d32078e1d4ace92f3e89c96071435 in term 0.
I20260928 11:02:44.693712 23774 leader_election.cc:302] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769051d203a345e4ae36556b7a01ea8a, e86d32078e1d4ace92f3e89c96071435; no voters:
I20260928 11:02:44.693768 23573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c04448fef4f52b7221b5ef09b9c13" candidate_uuid: "e86d32078e1d4ace92f3e89c96071435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" is_pre_election: true
I20260928 11:02:44.694087 23911 raft_consensus.cc:2781] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:02:44.694224 23573 raft_consensus.cc:2454] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e86d32078e1d4ace92f3e89c96071435 in term 0.
I20260928 11:02:44.694305 23910 leader_election.cc:288] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769051d203a345e4ae36556b7a01ea8a (127.3.252.66:41369), e86d32078e1d4ace92f3e89c96071435 (127.3.252.67:43983)
I20260928 11:02:44.694506 23911 raft_consensus.cc:482] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:02:44.694615 23911 raft_consensus.cc:3037] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:44.698107 23911 raft_consensus.cc:504] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.698768 23911 leader_election.cc:288] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [CANDIDATE]: Term 1 election: Requested vote from peers 769051d203a345e4ae36556b7a01ea8a (127.3.252.66:41369), 6744afeda3b14eeb80cd4e09a5ac7a50 (127.3.252.65:41967)
I20260928 11:02:44.699317 23573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c04448fef4f52b7221b5ef09b9c13" candidate_uuid: "e86d32078e1d4ace92f3e89c96071435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50"
I20260928 11:02:44.699522 23573 raft_consensus.cc:3037] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:44.700522 23707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c04448fef4f52b7221b5ef09b9c13" candidate_uuid: "e86d32078e1d4ace92f3e89c96071435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769051d203a345e4ae36556b7a01ea8a"
I20260928 11:02:44.700730 23707 raft_consensus.cc:3037] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:44.702651 23573 raft_consensus.cc:2454] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e86d32078e1d4ace92f3e89c96071435 in term 1.
I20260928 11:02:44.702760 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.695439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:37153 (local address 127.3.252.67:43983)
0928 11:02:44.695622 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.695628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.695769 (+ 141us) server_negotiation.cc:407] Connection header received
0928 11:02:44.695986 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.695994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.696066 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.696154 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:44.696898 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.697967 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.700329 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.701480 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.701638 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.701703 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.702385 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.702517 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 11:02:44.702890 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.696434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:35513 (local address 127.3.252.66:41369)
0928 11:02:44.696607 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.696613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.696629 (+ 16us) server_negotiation.cc:407] Connection header received
0928 11:02:44.696682 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.696687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.696771 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.696865 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:44.697327 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.698103 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.700220 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.701480 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.701633 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.701703 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.702385 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.702521 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 11:02:44.703104 23775 leader_election.cc:302] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6744afeda3b14eeb80cd4e09a5ac7a50, e86d32078e1d4ace92f3e89c96071435; no voters:
I20260928 11:02:44.703461 23911 raft_consensus.cc:2781] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:44.703593 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.694496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:41369 (local address 127.3.252.65:35513)
0928 11:02:44.695345 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.695366 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.695496 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.696919 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.696923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.696945 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:44.697200 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.697207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.698302 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.698308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.700074 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.700085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.701926 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.701931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.702149 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.702235 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.703079 (+ 844us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.703248 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":39,"thread_start_us":1686,"threads_started":1}
I20260928 11:02:44.703704 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.695000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:43983 (local address 127.3.252.65:37153)
0928 11:02:44.695673 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:44.695688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:44.695856 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:44.696422 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:44.696426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:44.696449 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:44.696744 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.696755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.698085 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.698090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.700187 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:44.700201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.701900 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.701911 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:44.702144 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.702239 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 11:02:44.702977 (+ 738us) client_negotiation.cc:240] Negotiation successful
0928 11:02:44.703183 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":74,"thread_start_us":624,"threads_started":1}
I20260928 11:02:44.704366 23706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c04448fef4f52b7221b5ef09b9c13" candidate_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769051d203a345e4ae36556b7a01ea8a" is_pre_election: true
I20260928 11:02:44.704582 23911 raft_consensus.cc:686] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 1 LEADER]: Becoming Leader. State: Replica: e86d32078e1d4ace92f3e89c96071435, State: Running, Role: LEADER
I20260928 11:02:44.705147 23911 consensus_queue.cc:237] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } }
I20260928 11:02:44.705335 23840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c04448fef4f52b7221b5ef09b9c13" candidate_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86d32078e1d4ace92f3e89c96071435" is_pre_election: true
I20260928 11:02:44.704340 23707 raft_consensus.cc:2454] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e86d32078e1d4ace92f3e89c96071435 in term 1.
I20260928 11:02:44.706329 23508 leader_election.cc:302] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6744afeda3b14eeb80cd4e09a5ac7a50; no voters: 769051d203a345e4ae36556b7a01ea8a, e86d32078e1d4ace92f3e89c96071435
I20260928 11:02:44.706730 23910 raft_consensus.cc:2726] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 11:02:44.710180 23429 catalog_manager.cc:5495] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 reported cstate change: term changed from 0 to 1, leader changed from <none> to e86d32078e1d4ace92f3e89c96071435 (127.3.252.67). New cstate: current_term: 1 leader_uuid: "e86d32078e1d4ace92f3e89c96071435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e86d32078e1d4ace92f3e89c96071435" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43983 } health_report { overall_health: HEALTHY } } }
I20260928 11:02:44.754272 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.748893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.3.252.67:43983)
0928 11:02:44.749068 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.749075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.749149 (+ 74us) server_negotiation.cc:407] Connection header received
0928 11:02:44.749386 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.749390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.749480 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.749610 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:02:44.750011 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.750738 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.752102 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.752524 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.752666 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.753164 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:02:44.753664 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:02:44.753802 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.753910 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.753992 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 11:02:44.762331 23707 raft_consensus.cc:1261] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 1 FOLLOWER]: Refusing update from remote peer e86d32078e1d4ace92f3e89c96071435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:44.762414 23573 raft_consensus.cc:1261] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 1 FOLLOWER]: Refusing update from remote peer e86d32078e1d4ace92f3e89c96071435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:44.763257 23915 consensus_queue.cc:1035] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769051d203a345e4ae36556b7a01ea8a" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:44.763638 23911 consensus_queue.cc:1035] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6744afeda3b14eeb80cd4e09a5ac7a50" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 41967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:44.778765 23918 mvcc.cc:205] Tried to move back new op lower bound from 7334270422057385984 to 7334270421836967936. Current Snapshot: MvccSnapshot[applied={T|T < 7334270422057385984}]
I20260928 11:02:44.799253 23429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:44.799571 23429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:44.803407 23429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 432a362a7bdf43debb3c90fae21a4c2f: Sending DeleteTablet for 3 replicas of tablet 9f2c04448fef4f52b7221b5ef09b9c13
I20260928 11:02:44.804762 23687 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2c04448fef4f52b7221b5ef09b9c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:44 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:60886
I20260928 11:02:44.804774 23820 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2c04448fef4f52b7221b5ef09b9c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:44 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:51382
I20260928 11:02:44.804739 23553 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2c04448fef4f52b7221b5ef09b9c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:44 UTC) from {username='kudu', principal='kudu/127.3.252.126@KRBTEST.COM'} at 127.0.0.1:35452
I20260928 11:02:44.806474 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 23490
I20260928 11:02:44.813570 23928 tablet_replica.cc:330] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: stopping tablet replica
I20260928 11:02:44.814126 23928 raft_consensus.cc:2229] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:44.814689 23928 raft_consensus.cc:2258] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:44.815653 23930 tablet_replica.cc:330] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: stopping tablet replica
I20260928 11:02:44.815690 23929 tablet_replica.cc:330] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: stopping tablet replica
I20260928 11:02:44.816064 23929 raft_consensus.cc:2229] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:02:44.816068 23930 raft_consensus.cc:2229] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:44.816601 23930 raft_consensus.cc:2258] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:44.816817 23929 raft_consensus.cc:2258] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:44.818830 23929 ts_tablet_manager.cc:1905] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:44.818826 23930 ts_tablet_manager.cc:1905] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:44.822285 23928 ts_tablet_manager.cc:1905] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:44.825973 23930 ts_tablet_manager.cc:1918] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:44.826135 23930 log.cc:1196] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-0/wal/wals/9f2c04448fef4f52b7221b5ef09b9c13
I20260928 11:02:44.826193 23929 ts_tablet_manager.cc:1918] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:44.826344 23929 log.cc:1196] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/9f2c04448fef4f52b7221b5ef09b9c13
I20260928 11:02:44.826701 23930 ts_tablet_manager.cc:1939] T 9f2c04448fef4f52b7221b5ef09b9c13 P 6744afeda3b14eeb80cd4e09a5ac7a50: Deleting consensus metadata
I20260928 11:02:44.826805 23929 ts_tablet_manager.cc:1939] T 9f2c04448fef4f52b7221b5ef09b9c13 P e86d32078e1d4ace92f3e89c96071435: Deleting consensus metadata
I20260928 11:02:44.828500 23415 catalog_manager.cc:4841] TS 6744afeda3b14eeb80cd4e09a5ac7a50 (127.3.252.65:41967): tablet 9f2c04448fef4f52b7221b5ef09b9c13 (table test-table [id=c21253d924cf4f97ac04a4ed29c1f0d7]) successfully deleted
I20260928 11:02:44.828864 23415 catalog_manager.cc:4841] TS e86d32078e1d4ace92f3e89c96071435 (127.3.252.67:43983): tablet 9f2c04448fef4f52b7221b5ef09b9c13 (table test-table [id=c21253d924cf4f97ac04a4ed29c1f0d7]) successfully deleted
I20260928 11:02:44.828933 23928 ts_tablet_manager.cc:1918] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:44.829098 23928 log.cc:1196] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/9f2c04448fef4f52b7221b5ef09b9c13
I20260928 11:02:44.829609 23928 ts_tablet_manager.cc:1939] T 9f2c04448fef4f52b7221b5ef09b9c13 P 769051d203a345e4ae36556b7a01ea8a: Deleting consensus metadata
Sep 28 11:02:44 dist-test-slave-djjw krb5kdc[23384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.65@KRBTEST.COM
I20260928 11:02:44.831378 23414 catalog_manager.cc:4841] TS 769051d203a345e4ae36556b7a01ea8a (127.3.252.66:41369): tablet 9f2c04448fef4f52b7221b5ef09b9c13 (table test-table [id=c21253d924cf4f97ac04a4ed29c1f0d7]) successfully deleted
I20260928 11:02:44.838680 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.807948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.3.252.65:41967)
0928 11:02:44.808133 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.808140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.813754 (+ 5614us) server_negotiation.cc:407] Connection header received
0928 11:02:44.814072 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:44.814079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:44.814168 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:44.814239 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:44.816070 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.817288 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.819179 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:44.819837 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:44.819945 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:44.832165 (+ 12220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:44.832181 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:44.832201 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:44.832247 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:44.835791 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:44.836552 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:44.836561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:44.836565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:44.836719 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:44.837042 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:44.837047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:44.837050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:44.837505 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:44.837801 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:44.838079 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 11:02:44.838318 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 11:02:44.839350 23613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:44.976723 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 23490
I20260928 11:02:44.998893 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 23624
Sep 28 11:02:45 dist-test-slave-djjw krb5kdc[23384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.66@KRBTEST.COM
I20260928 11:02:45.016041 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:44.999700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.3.252.66:41369)
0928 11:02:44.999890 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:44.999896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:44.999915 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:02:45.000006 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:45.000012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:45.000081 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:45.000165 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:45.001437 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:45.002719 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:45.003990 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:45.004690 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:45.004817 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:45.009154 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:45.009177 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:45.009200 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:45.009260 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:45.012966 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:45.013724 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:45.013735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:45.013739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:45.013929 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:45.014380 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:45.014386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:45.014389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:45.014865 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:45.015154 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:45.015417 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 11:02:45.015650 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:02:45.016754 23747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:45.105595 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 23624
I20260928 11:02:45.126592 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 23757
Sep 28 11:02:45 dist-test-slave-djjw krb5kdc[23384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.67@KRBTEST.COM
I20260928 11:02:45.142444 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:45.127412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.3.252.67:43983)
0928 11:02:45.127575 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:45.127581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:45.127655 (+ 74us) server_negotiation.cc:407] Connection header received
0928 11:02:45.127844 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:45.127850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:45.127944 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:45.128048 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:45.129496 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:45.130335 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:45.131604 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:45.132037 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:45.132152 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:45.136676 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:45.136693 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:45.136710 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:45.136753 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:45.139670 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:45.140314 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:45.140322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:45.140324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:45.140462 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:45.140861 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:45.140865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:45.140867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:45.141257 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:45.141526 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:45.141890 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 11:02:45.142116 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 11:02:45.143052 23880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:45.264386 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 23757
I20260928 11:02:45.280954 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 23397
I20260928 11:02:45.282126 23459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:45.395726 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 23397
2026-09-28T11:02:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4317 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T11:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:45Z Disabled control of system clock
I20260928 11:02:45.466419 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:39043
--webserver_interface=127.3.252.126
--webserver_port=0
--builtin_ntp_servers=127.3.252.84:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:39043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 11:02:45.789587 23939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:45.789904 23939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:45.790032 23939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:45.797315 23939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:45.797434 23939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:45.797499 23939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:45.797580 23939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:45.797638 23939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:45.806983 23939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:39043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:39043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:45.808245 23939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:45.809957 23939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:45.820962 23945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:45.821018 23944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:45.821626 23939 server_base.cc:1034] running on GCE node
W20260928 11:02:45.821646 23947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:45.822434 23939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:45.823817 23939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:45.825029 23939 hybrid_clock.cc:648] HybridClock initialized: now 1790593365824990 us; error 48 us; skew 500 ppm
I20260928 11:02:45.825378 23939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:45.827517 23939 webserver.cc:460] Webserver started at http://127.3.252.126:38825/ using document root <none> and password file <none>
I20260928 11:02:45.828204 23939 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:45.828303 23939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:45.828596 23939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:45.831456 23939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "e30baa6bcb824f1984d82a045e1c5ad7"
format_stamp: "Formatted at 2026-09-28 11:02:45 on dist-test-slave-djjw"
server_key: "e3f4a71c31498b462da8ebf181acc4dd"
server_key_iv: "590a7cbcf871a25adf78b9e3374b2dee"
server_key_version: "encryptionkey@0"
I20260928 11:02:45.832226 23939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "e30baa6bcb824f1984d82a045e1c5ad7"
format_stamp: "Formatted at 2026-09-28 11:02:45 on dist-test-slave-djjw"
server_key: "e3f4a71c31498b462da8ebf181acc4dd"
server_key_iv: "590a7cbcf871a25adf78b9e3374b2dee"
server_key_version: "encryptionkey@0"
I20260928 11:02:45.837306 23939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 11:02:45.840718 23953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:45.842062 23939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:45.842264 23939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "e30baa6bcb824f1984d82a045e1c5ad7"
format_stamp: "Formatted at 2026-09-28 11:02:45 on dist-test-slave-djjw"
server_key: "e3f4a71c31498b462da8ebf181acc4dd"
server_key_iv: "590a7cbcf871a25adf78b9e3374b2dee"
server_key_version: "encryptionkey@0"
I20260928 11:02:45.842468 23939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:45.863444 23939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:45.864364 23939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:45.864642 23939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:45.881259 23939 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:39043
I20260928 11:02:45.881309 24004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:39043 every 8 connection(s)
I20260928 11:02:45.882601 23939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:45.886426 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:45.889343 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 23939
I20260928 11:02:45.889505 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:45.890056 4081 external_mini_cluster.cc:1428] Setting key c9de8d361b63a16c0782c1dbab86eef7
I20260928 11:02:45.898398 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7: Bootstrap starting.
I20260928 11:02:45.901630 24005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:45.902750 24005 log.cc:824] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:45.903462 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:45.892093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.3.252.126:39043)
0928 11:02:45.893125 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:45.893136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:45.893208 (+ 72us) server_negotiation.cc:407] Connection header received
0928 11:02:45.894937 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:45.894964 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:45.895469 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:45.895854 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:45.896425 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:45.897655 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:45.899036 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:45.899722 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:45.899895 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:45.901005 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:45.901012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:45.901026 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:45.901576 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:45.901668 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:45.901700 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:45.901803 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:45.902065 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 11:02:45.902415 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":377,"threads_started":1}
I20260928 11:02:45.905814 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7: No bootstrap required, opened a new log
I20260928 11:02:45.913268 24005 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39043 } }
I20260928 11:02:45.913671 24005 raft_consensus.cc:374] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:45.913765 24005 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30baa6bcb824f1984d82a045e1c5ad7, State: Initialized, Role: FOLLOWER
I20260928 11:02:45.914353 24005 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39043 } }
I20260928 11:02:45.914577 24005 raft_consensus.cc:388] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:45.914712 24005 raft_consensus.cc:482] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:45.914855 24005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:45.916988 24005 raft_consensus.cc:504] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39043 } }
I20260928 11:02:45.917469 24005 leader_election.cc:302] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e30baa6bcb824f1984d82a045e1c5ad7; no voters:
I20260928 11:02:45.918092 24005 leader_election.cc:288] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:45.918263 24010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:45.919283 24010 raft_consensus.cc:686] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [term 1 LEADER]: Becoming Leader. State: Replica: e30baa6bcb824f1984d82a045e1c5ad7, State: Running, Role: LEADER
I20260928 11:02:45.919775 24010 consensus_queue.cc:237] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39043 } }
I20260928 11:02:45.920193 24005 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:45.922256 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39043 } } }
I20260928 11:02:45.922648 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:45.922561 24012 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30baa6bcb824f1984d82a045e1c5ad7. Latest consensus state: current_term: 1 leader_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30baa6bcb824f1984d82a045e1c5ad7" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 39043 } } }
I20260928 11:02:45.923158 24012 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:45.925504 24019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:45.930572 24019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:45.940011 24019 catalog_manager.cc:1347] Generated new cluster ID: afe86982186b4197b0a66da0713b757d
I20260928 11:02:45.940146 24019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:45.977015 24019 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:45.978878 24019 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:45.990322 24019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7: Generated new TSK 0
I20260928 11:02:45.991365 24019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:46.001685 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.65:0
--local_ip_for_outbound_sockets=127.3.252.65
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39043
--builtin_ntp_servers=127.3.252.84:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 11:02:46.328603 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:46.328898 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:46.329022 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:46.336562 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:46.336673 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:46.336824 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.65
I20260928 11:02:46.346294 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.252.65
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.65
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:46.347631 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:46.349434 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:46.363278 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:46.363338 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:46.363322 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:46.364152 24029 server_base.cc:1034] running on GCE node
I20260928 11:02:46.364742 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:46.365712 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:46.366926 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790593366366821 us; error 99 us; skew 500 ppm
I20260928 11:02:46.367252 24029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:46.369560 24029 webserver.cc:460] Webserver started at http://127.3.252.65:39045/ using document root <none> and password file <none>
I20260928 11:02:46.370302 24029 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:46.370425 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:46.370744 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:46.373644 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data/instance:
uuid: "0993fbc4fe434497aa83af6dbb586f00"
format_stamp: "Formatted at 2026-09-28 11:02:46 on dist-test-slave-djjw"
server_key: "13ce9158826a85d4b4d968f43d458faa"
server_key_iv: "08283c2fbf080c9735a75c0facc62ac0"
server_key_version: "encryptionkey@0"
I20260928 11:02:46.374465 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance:
uuid: "0993fbc4fe434497aa83af6dbb586f00"
format_stamp: "Formatted at 2026-09-28 11:02:46 on dist-test-slave-djjw"
server_key: "13ce9158826a85d4b4d968f43d458faa"
server_key_iv: "08283c2fbf080c9735a75c0facc62ac0"
server_key_version: "encryptionkey@0"
I20260928 11:02:46.379535 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:02:46.383090 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:46.384455 24029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 11:02:46.384609 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/wal
uuid: "0993fbc4fe434497aa83af6dbb586f00"
format_stamp: "Formatted at 2026-09-28 11:02:46 on dist-test-slave-djjw"
server_key: "13ce9158826a85d4b4d968f43d458faa"
server_key_iv: "08283c2fbf080c9735a75c0facc62ac0"
server_key_version: "encryptionkey@0"
I20260928 11:02:46.384812 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:46.410162 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:46.411038 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:46.411329 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:46.412338 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:46.413869 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:46.413985 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:46.414086 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:46.414227 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:46.444674 24029 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.65:43605
I20260928 11:02:46.444739 24155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.65:43605 every 8 connection(s)
I20260928 11:02:46.446054 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/data/info.pb
I20260928 11:02:46.455816 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24029
I20260928 11:02:46.455977 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-0/wal/instance
I20260928 11:02:46.456399 4081 external_mini_cluster.cc:1428] Setting key 39e4bb72a840affe9ef342de176fa580
I20260928 11:02:46.458348 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:46.448566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.65:58003 (local address 127.3.252.126:39043)
0928 11:02:46.448956 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:46.448961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:46.450159 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 11:02:46.451939 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:46.451945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:46.452027 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:46.452120 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:46.453181 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.453944 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.455461 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.455869 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.455979 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:46.457502 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:46.457507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:46.457510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:46.457528 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:46.457583 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:46.457596 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:46.457668 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:46.457911 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 11:02:46.458012 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":177,"threads_started":1}
I20260928 11:02:46.459041 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.66:0
--local_ip_for_outbound_sockets=127.3.252.66
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39043
--builtin_ntp_servers=127.3.252.84:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 11:02:46.459416 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:46.449118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39043 (local address 127.3.252.65:58003)
0928 11:02:46.450022 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:46.450058 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:46.451670 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:46.452351 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:46.452359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:46.452451 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:46.453017 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:46.453034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.454087 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.454090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:46.455319 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:46.455332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.456091 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.456099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:46.456317 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:46.457189 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:46.457192 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:46.457233 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:46.457322 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:46.457682 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:46.457695 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:46.457702 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 11:02:46.457902 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 11:02:46.458320 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":243,"threads_started":1}
I20260928 11:02:46.461045 24156 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39043
I20260928 11:02:46.461344 24156 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:46.462128 24156 heartbeater.cc:511] Master 127.3.252.126:39043 requested a full tablet report, sending...
I20260928 11:02:46.464421 23970 ts_manager.cc:194] Registered new tserver with Master: 0993fbc4fe434497aa83af6dbb586f00 (127.3.252.65:43605)
I20260928 11:02:46.466313 23970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.65:58003
W20260928 11:02:46.778517 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:46.778805 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:46.778908 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:46.786231 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:46.786346 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:46.786485 24160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.66
I20260928 11:02:46.795749 24160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.252.66
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.66
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:46.797050 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:46.798828 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:46.811522 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:46.811563 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:46.811573 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:46.812580 24160 server_base.cc:1034] running on GCE node
I20260928 11:02:46.813112 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:46.814033 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:46.815220 24160 hybrid_clock.cc:648] HybridClock initialized: now 1790593366815167 us; error 44 us; skew 500 ppm
I20260928 11:02:46.815547 24160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:46.817786 24160 webserver.cc:460] Webserver started at http://127.3.252.66:42021/ using document root <none> and password file <none>
I20260928 11:02:46.818468 24160 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:46.818570 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:46.818867 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:46.821727 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data/instance:
uuid: "87d1d270fb7343aa864ea4550c51f519"
format_stamp: "Formatted at 2026-09-28 11:02:46 on dist-test-slave-djjw"
server_key: "0ac68d0341307e3e4c6a1158afed75cc"
server_key_iv: "7c950466b593b392332f616b32fa5ee6"
server_key_version: "encryptionkey@0"
I20260928 11:02:46.822491 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance:
uuid: "87d1d270fb7343aa864ea4550c51f519"
format_stamp: "Formatted at 2026-09-28 11:02:46 on dist-test-slave-djjw"
server_key: "0ac68d0341307e3e4c6a1158afed75cc"
server_key_iv: "7c950466b593b392332f616b32fa5ee6"
server_key_version: "encryptionkey@0"
I20260928 11:02:46.827445 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 11:02:46.830688 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:46.832041 24160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:02:46.832201 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal
uuid: "87d1d270fb7343aa864ea4550c51f519"
format_stamp: "Formatted at 2026-09-28 11:02:46 on dist-test-slave-djjw"
server_key: "0ac68d0341307e3e4c6a1158afed75cc"
server_key_iv: "7c950466b593b392332f616b32fa5ee6"
server_key_version: "encryptionkey@0"
I20260928 11:02:46.832412 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:46.856029 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:46.856876 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:46.857161 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:46.858108 24160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:46.859532 24160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:46.859617 24160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:46.859720 24160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:46.859784 24160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:46.889518 24160 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.66:36993
I20260928 11:02:46.889570 24286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.66:36993 every 8 connection(s)
I20260928 11:02:46.890794 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/data/info.pb
I20260928 11:02:46.894049 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24160
I20260928 11:02:46.894205 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal/instance
I20260928 11:02:46.894608 4081 external_mini_cluster.cc:1428] Setting key 20eca7296b1a541466403b7285c75fe6
I20260928 11:02:46.897589 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.252.67:0
--local_ip_for_outbound_sockets=127.3.252.67
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39043
--builtin_ntp_servers=127.3.252.84:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 11:02:46.904906 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:46.893317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:41799 (local address 127.3.252.126:39043)
0928 11:02:46.893480 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:46.893485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:46.895961 (+ 2476us) server_negotiation.cc:407] Connection header received
0928 11:02:46.898283 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:46.898289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:46.898366 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:46.898467 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:46.899734 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.900408 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.902361 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.902747 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.902837 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:46.904270 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:46.904275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:46.904278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:46.904296 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:46.904328 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:46.904339 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:46.904406 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:46.904609 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 11:02:46.904686 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 11:02:46.906214 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:46.893979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39043 (local address 127.3.252.66:41799)
0928 11:02:46.895806 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:46.895848 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:46.897992 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:46.898700 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:46.898712 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:46.898849 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:46.899566 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:46.899589 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.900547 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.900552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:46.902233 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:46.902247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:46.902955 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:46.902962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:46.903227 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:46.903914 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:46.903917 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:46.903966 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:46.904066 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:46.904422 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:46.904431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:46.904439 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 11:02:46.904606 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 11:02:46.904983 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"thread_start_us":1136,"threads_started":1}
I20260928 11:02:46.907650 24287 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39043
I20260928 11:02:46.907948 24287 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:46.908649 24287 heartbeater.cc:511] Master 127.3.252.126:39043 requested a full tablet report, sending...
I20260928 11:02:46.910084 23970 ts_manager.cc:194] Registered new tserver with Master: 87d1d270fb7343aa864ea4550c51f519 (127.3.252.66:36993)
I20260928 11:02:46.911064 23970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.66:41799
W20260928 11:02:47.236984 24290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:47.237284 24290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:47.237370 24290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:47.244645 24290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:47.244752 24290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:47.244859 24290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.252.67
I20260928 11:02:47.254492 24290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.252.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.252.67
--webserver_port=0
--tserver_master_addrs=127.3.252.126:39043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.252.67
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:47.256093 24290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:47.257979 24290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:47.271348 24295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:47.271308 24298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:47.271265 24296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:47.273186 24290 server_base.cc:1034] running on GCE node
I20260928 11:02:47.273851 24290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:47.274849 24290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:47.276068 24290 hybrid_clock.cc:648] HybridClock initialized: now 1790593367276020 us; error 40 us; skew 500 ppm
I20260928 11:02:47.276415 24290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 11:02:47.278743 24290 webserver.cc:460] Webserver started at http://127.3.252.67:38603/ using document root <none> and password file <none>
I20260928 11:02:47.279443 24290 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:47.279543 24290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:47.279845 24290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:47.282773 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data/instance:
uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0"
format_stamp: "Formatted at 2026-09-28 11:02:47 on dist-test-slave-djjw"
server_key: "a24256fd95b5f7d72e967bb29d8af3de"
server_key_iv: "2fa2893c8affeb72acc1b0bc4357622c"
server_key_version: "encryptionkey@0"
I20260928 11:02:47.283567 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance:
uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0"
format_stamp: "Formatted at 2026-09-28 11:02:47 on dist-test-slave-djjw"
server_key: "a24256fd95b5f7d72e967bb29d8af3de"
server_key_iv: "2fa2893c8affeb72acc1b0bc4357622c"
server_key_version: "encryptionkey@0"
I20260928 11:02:47.288973 24290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:02:47.292912 24304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:47.294503 24290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:47.294677 24290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal
uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0"
format_stamp: "Formatted at 2026-09-28 11:02:47 on dist-test-slave-djjw"
server_key: "a24256fd95b5f7d72e967bb29d8af3de"
server_key_iv: "2fa2893c8affeb72acc1b0bc4357622c"
server_key_version: "encryptionkey@0"
I20260928 11:02:47.294901 24290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:47.316975 24290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:47.318034 24290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:47.318398 24290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:47.319417 24290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 11:02:47.320982 24290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 11:02:47.321079 24290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:47.321179 24290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 11:02:47.321252 24290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:47.352571 24290 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.67:43045
I20260928 11:02:47.352635 24416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.67:43045 every 8 connection(s)
I20260928 11:02:47.353813 24290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/data/info.pb
I20260928 11:02:47.356801 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24290
I20260928 11:02:47.356966 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal/instance
I20260928 11:02:47.357446 4081 external_mini_cluster.cc:1428] Setting key 88687cd7bf9fddfd04bc5198b7a0d9f4
I20260928 11:02:47.366341 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.356274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.67:34457 (local address 127.3.252.126:39043)
0928 11:02:47.356504 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.356511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.357970 (+ 1459us) server_negotiation.cc:407] Connection header received
0928 11:02:47.360178 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.360183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.360299 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.360403 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:47.361523 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.362340 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.363875 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.364282 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.364373 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.365642 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:47.365646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:47.365649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:47.365669 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:47.365704 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:47.365716 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:47.365793 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.366007 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.366092 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 11:02:47.367419 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.356753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:39043 (local address 127.3.252.67:34457)
0928 11:02:47.357773 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.357823 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:47.359933 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:47.360629 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:47.360638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:47.360731 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:02:47.361352 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.361370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.362494 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.362499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.363749 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.363761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.364450 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.364454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.364659 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.365317 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 11:02:47.365319 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:02:47.365362 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 11:02:47.365454 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:02:47.365788 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:02:47.365793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:02:47.365806 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 11:02:47.365989 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 11:02:47.366345 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":326,"threads_started":1}
I20260928 11:02:47.368714 24417 heartbeater.cc:347] Connected to a master server at 127.3.252.126:39043
I20260928 11:02:47.368997 24417 heartbeater.cc:464] Registering TS with master...
I20260928 11:02:47.369699 24417 heartbeater.cc:511] Master 127.3.252.126:39043 requested a full tablet report, sending...
I20260928 11:02:47.371109 23970 ts_manager.cc:194] Registered new tserver with Master: c375d460d3cb4f19ad6a2e2de4d6ccc0 (127.3.252.67:43045)
I20260928 11:02:47.371886 23970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.252.67:34457
I20260928 11:02:47.372578 4081 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 11:02:47.381978 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.376488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.3.252.126:39043)
0928 11:02:47.376681 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.376687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.377005 (+ 318us) server_negotiation.cc:407] Connection header received
0928 11:02:47.377171 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.377176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.377245 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.377333 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:47.377794 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.378522 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.379850 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.380244 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.380356 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.381294 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:47.381299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:47.381302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:47.381319 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:47.381349 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:47.381360 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:47.381430 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.381553 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.381639 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 11:02:47.385552 23969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 11:02:47.392536 23969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 11:02:47.429380 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.404394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.3.252.65:43605)
0928 11:02:47.419923 (+ 15529us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.419931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.420693 (+ 762us) server_negotiation.cc:407] Connection header received
0928 11:02:47.420894 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.420904 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.421173 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.421354 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:47.422430 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.423971 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.426291 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.427771 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.427905 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.428158 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.428823 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.429009 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15391,"thread_start_us":15284,"threads_started":1}
I20260928 11:02:47.429510 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.406224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:43045 (local address 127.0.0.1:45480)
0928 11:02:47.421105 (+ 14881us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.421122 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:47.421228 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:47.422044 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:47.422049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:47.422075 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:47.422537 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.422589 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.424371 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.424375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.426206 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.426221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.428323 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.428328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.428533 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.428603 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 11:02:47.428745 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 11:02:47.429021 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14757,"mutex_wait_us":116,"thread_start_us":14192,"threads_started":1}
I20260928 11:02:47.429559 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.404156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60168)
0928 11:02:47.420529 (+ 16373us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.420548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:47.420690 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:47.421652 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:47.421659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:47.421711 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:47.422236 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.422251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.424126 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.424134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.426091 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.426105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.428314 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.428321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.428527 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.428603 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 11:02:47.428737 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 11:02:47.429021 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16215,"mutex_wait_us":126,"spinlock_wait_cycles":2304,"thread_start_us":16733,"threads_started":1}
I20260928 11:02:47.430253 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.406716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.3.252.67:43045)
0928 11:02:47.420570 (+ 13854us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.420579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.421343 (+ 764us) server_negotiation.cc:407] Connection header received
0928 11:02:47.421564 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.421621 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.421854 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.422033 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:47.422757 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.424219 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.426388 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.427781 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.427905 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.428192 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.429716 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.429912 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13653,"thread_start_us":13502,"threads_started":1}
I20260928 11:02:47.431316 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.403981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.66:36993 (local address 127.0.0.1:37418)
0928 11:02:47.419244 (+ 15263us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.419300 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:47.419539 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:47.421652 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:47.421659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:47.421701 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:47.422318 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.422330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.423846 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.423850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.427058 (+ 3208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.427070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.429091 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.429097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.429270 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.429305 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 11:02:47.429393 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 11:02:47.431074 (+ 1681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14922,"mutex_wait_us":1063,"spinlock_wait_cycles":2944,"thread_start_us":15780,"threads_started":1}
I20260928 11:02:47.431703 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.404235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.3.252.66:36993)
0928 11:02:47.419474 (+ 15239us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.419485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.419529 (+ 44us) server_negotiation.cc:407] Connection header received
0928 11:02:47.421208 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.421217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.421447 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.421664 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:47.422531 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.423692 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.427251 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.428583 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.428729 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.428962 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.430063 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.430235 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15032,"thread_start_us":15684,"threads_started":1}
I20260928 11:02:47.433682 24091 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc4e6113484408eb73f6d68b37795b8 (DEFAULT_TABLE table=test-table [id=c215e4e5399846bfa31ed8d1856116a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:47.433682 24352 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc4e6113484408eb73f6d68b37795b8 (DEFAULT_TABLE table=test-table [id=c215e4e5399846bfa31ed8d1856116a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:47.434798 24222 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc4e6113484408eb73f6d68b37795b8 (DEFAULT_TABLE table=test-table [id=c215e4e5399846bfa31ed8d1856116a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 11:02:47.435196 24352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc4e6113484408eb73f6d68b37795b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:47.435196 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc4e6113484408eb73f6d68b37795b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:47.435933 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc4e6113484408eb73f6d68b37795b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:47.452868 24436 tablet_bootstrap.cc:492] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: Bootstrap starting.
I20260928 11:02:47.457252 24436 tablet_bootstrap.cc:654] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:47.459578 24435 tablet_bootstrap.cc:492] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Bootstrap starting.
I20260928 11:02:47.461483 24436 log.cc:824] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:47.462154 24437 tablet_bootstrap.cc:492] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Bootstrap starting.
I20260928 11:02:47.463150 24435 tablet_bootstrap.cc:654] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:47.464407 24435 log.cc:824] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:47.464934 24436 tablet_bootstrap.cc:492] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: No bootstrap required, opened a new log
I20260928 11:02:47.465241 24436 ts_tablet_manager.cc:1397] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260928 11:02:47.465284 24437 tablet_bootstrap.cc:654] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:47.466434 24437 log.cc:824] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:47.469014 24437 tablet_bootstrap.cc:492] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: No bootstrap required, opened a new log
I20260928 11:02:47.469239 24435 tablet_bootstrap.cc:492] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: No bootstrap required, opened a new log
I20260928 11:02:47.469316 24156 heartbeater.cc:503] Master 127.3.252.126:39043 was elected leader, sending a full tablet report...
I20260928 11:02:47.469543 24435 ts_tablet_manager.cc:1397] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 11:02:47.470091 24437 ts_tablet_manager.cc:1397] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 11:02:47.475972 24436 raft_consensus.cc:348] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.476468 24436 raft_consensus.cc:374] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:47.476567 24436 raft_consensus.cc:729] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0993fbc4fe434497aa83af6dbb586f00, State: Initialized, Role: FOLLOWER
I20260928 11:02:47.477214 24436 consensus_queue.cc:260] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.478756 24437 raft_consensus.cc:348] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.479123 24437 raft_consensus.cc:374] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:47.479216 24437 raft_consensus.cc:729] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c375d460d3cb4f19ad6a2e2de4d6ccc0, State: Initialized, Role: FOLLOWER
I20260928 11:02:47.479753 24437 consensus_queue.cc:260] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.479923 24435 raft_consensus.cc:348] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.480370 24435 raft_consensus.cc:374] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:47.480463 24435 raft_consensus.cc:729] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d1d270fb7343aa864ea4550c51f519, State: Initialized, Role: FOLLOWER
I20260928 11:02:47.480944 24417 heartbeater.cc:503] Master 127.3.252.126:39043 was elected leader, sending a full tablet report...
I20260928 11:02:47.481119 24435 consensus_queue.cc:260] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.481340 24437 ts_tablet_manager.cc:1428] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 11:02:47.482395 24287 heartbeater.cc:503] Master 127.3.252.126:39043 was elected leader, sending a full tablet report...
I20260928 11:02:47.482813 24435 ts_tablet_manager.cc:1428] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260928 11:02:47.484148 24436 ts_tablet_manager.cc:1428] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20260928 11:02:47.568168 24443 raft_consensus.cc:482] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 11:02:47.568619 24443 raft_consensus.cc:504] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.570881 24443 leader_election.cc:288] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0993fbc4fe434497aa83af6dbb586f00 (127.3.252.65:43605), c375d460d3cb4f19ad6a2e2de4d6ccc0 (127.3.252.67:43045)
I20260928 11:02:47.578361 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.571776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:36349 (local address 127.3.252.67:43045)
0928 11:02:47.571987 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.571992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.572012 (+ 20us) server_negotiation.cc:407] Connection header received
0928 11:02:47.572303 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.572309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.572390 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.572457 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:47.573269 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.574405 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.576585 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.577374 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.577501 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.577564 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.578000 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.578126 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 11:02:47.578390 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.571253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.252.66:37471 (local address 127.3.252.65:43605)
0928 11:02:47.571433 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.571438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.572321 (+ 883us) server_negotiation.cc:407] Connection header received
0928 11:02:47.572528 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.572534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.572608 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.572690 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 11:02:47.573382 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.574423 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.576592 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.577470 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.577620 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.577687 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.578022 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.578141 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 11:02:47.578729 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.571694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.3.252.66:37471)
0928 11:02:47.572204 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.572217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:47.572365 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:47.572848 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:47.572852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:47.572877 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:47.573240 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.573248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.574536 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.574540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.576380 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.576392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.577660 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.577664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.577792 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.577893 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 11:02:47.577970 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 11:02:47.578095 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":135,"thread_start_us":495,"threads_started":1}
I20260928 11:02:47.578773 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.571236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.67:43045 (local address 127.3.252.66:36349)
0928 11:02:47.571921 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.571943 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 11:02:47.572100 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:02:47.572650 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:02:47.572655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:02:47.572693 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 11:02:47.573113 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.573123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.574533 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.574538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.576373 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:02:47.576386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.577561 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.577568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:02:47.577781 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.577876 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 11:02:47.577957 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 11:02:47.578092 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":167,"thread_start_us":778,"threads_started":1}
I20260928 11:02:47.579921 24372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc4e6113484408eb73f6d68b37795b8" candidate_uuid: "87d1d270fb7343aa864ea4550c51f519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" is_pre_election: true
I20260928 11:02:47.579926 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc4e6113484408eb73f6d68b37795b8" candidate_uuid: "87d1d270fb7343aa864ea4550c51f519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0993fbc4fe434497aa83af6dbb586f00" is_pre_election: true
I20260928 11:02:47.580567 24372 raft_consensus.cc:2454] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d1d270fb7343aa864ea4550c51f519 in term 0.
I20260928 11:02:47.580567 24111 raft_consensus.cc:2454] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d1d270fb7343aa864ea4550c51f519 in term 0.
I20260928 11:02:47.581341 24177 leader_election.cc:302] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0993fbc4fe434497aa83af6dbb586f00, 87d1d270fb7343aa864ea4550c51f519; no voters:
I20260928 11:02:47.581681 24443 raft_consensus.cc:2781] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 11:02:47.581858 24443 raft_consensus.cc:482] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 11:02:47.581959 24443 raft_consensus.cc:3037] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:47.584132 24443 raft_consensus.cc:504] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.584681 24443 leader_election.cc:288] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [CANDIDATE]: Term 1 election: Requested vote from peers 0993fbc4fe434497aa83af6dbb586f00 (127.3.252.65:43605), c375d460d3cb4f19ad6a2e2de4d6ccc0 (127.3.252.67:43045)
I20260928 11:02:47.585103 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc4e6113484408eb73f6d68b37795b8" candidate_uuid: "87d1d270fb7343aa864ea4550c51f519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0993fbc4fe434497aa83af6dbb586f00"
I20260928 11:02:47.585115 24372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc4e6113484408eb73f6d68b37795b8" candidate_uuid: "87d1d270fb7343aa864ea4550c51f519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0"
I20260928 11:02:47.585283 24111 raft_consensus.cc:3037] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:47.585289 24372 raft_consensus.cc:3037] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:47.587466 24111 raft_consensus.cc:2454] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d1d270fb7343aa864ea4550c51f519 in term 1.
I20260928 11:02:47.587484 24372 raft_consensus.cc:2454] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d1d270fb7343aa864ea4550c51f519 in term 1.
I20260928 11:02:47.587829 24177 leader_election.cc:302] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0993fbc4fe434497aa83af6dbb586f00, 87d1d270fb7343aa864ea4550c51f519; no voters:
I20260928 11:02:47.588114 24443 raft_consensus.cc:2781] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:47.588553 24443 raft_consensus.cc:686] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 1 LEADER]: Becoming Leader. State: Replica: 87d1d270fb7343aa864ea4550c51f519, State: Running, Role: LEADER
I20260928 11:02:47.589051 24443 consensus_queue.cc:237] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } }
I20260928 11:02:47.594000 23969 catalog_manager.cc:5495] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d1d270fb7343aa864ea4550c51f519 (127.3.252.66). New cstate: current_term: 1 leader_uuid: "87d1d270fb7343aa864ea4550c51f519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d1d270fb7343aa864ea4550c51f519" member_type: VOTER last_known_addr { host: "127.3.252.66" port: 36993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 } health_report { overall_health: UNKNOWN } } }
W20260928 11:02:47.605429 24418 tablet.cc:2369] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:02:47.629220 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.624289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.3.252.66:36993)
0928 11:02:47.624475 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.624479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.624500 (+ 21us) server_negotiation.cc:407] Connection header received
0928 11:02:47.624717 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.624721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.624816 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.624939 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:02:47.625223 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.626002 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.627347 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.627744 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.627832 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.628229 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:02:47.628761 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:02:47.628863 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.628951 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.629016 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 11:02:47.637406 24372 raft_consensus.cc:1261] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 1 FOLLOWER]: Refusing update from remote peer 87d1d270fb7343aa864ea4550c51f519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:47.637393 24111 raft_consensus.cc:1261] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 1 FOLLOWER]: Refusing update from remote peer 87d1d270fb7343aa864ea4550c51f519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:02:47.638480 24446 consensus_queue.cc:1035] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c375d460d3cb4f19ad6a2e2de4d6ccc0" member_type: VOTER last_known_addr { host: "127.3.252.67" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:02:47.638804 24443 consensus_queue.cc:1035] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0993fbc4fe434497aa83af6dbb586f00" member_type: VOTER last_known_addr { host: "127.3.252.65" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 11:02:47.642771 24288 tablet.cc:2369] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:02:47.664274 23969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:47.664507 23969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 11:02:47.668056 23969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e30baa6bcb824f1984d82a045e1c5ad7: Sending DeleteTablet for 3 replicas of tablet 8fc4e6113484408eb73f6d68b37795b8
I20260928 11:02:47.669154 24352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc4e6113484408eb73f6d68b37795b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:47 UTC) from {username='slave'} at 127.0.0.1:45480
I20260928 11:02:47.669128 24222 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc4e6113484408eb73f6d68b37795b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:47 UTC) from {username='slave'} at 127.0.0.1:37418
I20260928 11:02:47.669194 24091 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc4e6113484408eb73f6d68b37795b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 11:02:47 UTC) from {username='slave'} at 127.0.0.1:60168
I20260928 11:02:47.672421 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24029
I20260928 11:02:47.682406 24461 tablet_replica.cc:330] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: stopping tablet replica
I20260928 11:02:47.682937 24461 raft_consensus.cc:2229] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:47.683516 24461 raft_consensus.cc:2258] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:47.688135 24461 ts_tablet_manager.cc:1905] T 8fc4e6113484408eb73f6d68b37795b8 P 0993fbc4fe434497aa83af6dbb586f00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:47.691354 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.673903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.3.252.65:43605)
0928 11:02:47.674084 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.674091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.682671 (+ 8580us) server_negotiation.cc:407] Connection header received
0928 11:02:47.682948 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.682954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.683060 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.683141 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:47.683994 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.685150 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.687053 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.687758 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.687856 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.689543 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:47.689555 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:47.689573 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:47.690407 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:47.690466 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:47.690492 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:47.690583 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.690832 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.690954 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 11:02:47.692251 24460 tablet_replica.cc:330] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: stopping tablet replica
I20260928 11:02:47.692771 24460 raft_consensus.cc:2229] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 11:02:47.692888 24151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:47.693413 24460 raft_consensus.cc:2258] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:47.694551 24459 tablet_replica.cc:330] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: stopping tablet replica
I20260928 11:02:47.695065 24459 raft_consensus.cc:2229] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 1 LEADER]: Raft consensus shutting down.
I20260928 11:02:47.695779 24459 raft_consensus.cc:2258] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 11:02:47.696064 24460 ts_tablet_manager.cc:1905] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:47.697924 24459 ts_tablet_manager.cc:1905] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 11:02:47.703644 24460 ts_tablet_manager.cc:1918] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:47.703826 24460 log.cc:1196] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-2/wal/wals/8fc4e6113484408eb73f6d68b37795b8
I20260928 11:02:47.704490 24460 ts_tablet_manager.cc:1939] T 8fc4e6113484408eb73f6d68b37795b8 P c375d460d3cb4f19ad6a2e2de4d6ccc0: Deleting consensus metadata
I20260928 11:02:47.705966 24459 ts_tablet_manager.cc:1918] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 11:02:47.706218 24459 log.cc:1196] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Deleting WAL directory at /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790593056029081-4081-0/minicluster-data/ts-1/wal/wals/8fc4e6113484408eb73f6d68b37795b8
I20260928 11:02:47.706362 23955 catalog_manager.cc:4841] TS c375d460d3cb4f19ad6a2e2de4d6ccc0 (127.3.252.67:43045): tablet 8fc4e6113484408eb73f6d68b37795b8 (table test-table [id=c215e4e5399846bfa31ed8d1856116a5]) successfully deleted
I20260928 11:02:47.706805 24459 ts_tablet_manager.cc:1939] T 8fc4e6113484408eb73f6d68b37795b8 P 87d1d270fb7343aa864ea4550c51f519: Deleting consensus metadata
I20260928 11:02:47.708531 23957 catalog_manager.cc:4841] TS 87d1d270fb7343aa864ea4550c51f519 (127.3.252.66:36993): tablet 8fc4e6113484408eb73f6d68b37795b8 (table test-table [id=c215e4e5399846bfa31ed8d1856116a5]) successfully deleted
I20260928 11:02:47.790865 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24029
W20260928 11:02:47.811297 23956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.252.65:43605 (error 108)
I20260928 11:02:47.811906 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24160
I20260928 11:02:47.813552 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.813047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60184)
0928 11:02:47.813271 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.813388 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260928 11:02:47.813782 23956 catalog_manager.cc:4584] TS 0993fbc4fe434497aa83af6dbb586f00 (127.3.252.65:43605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fc4e6113484408eb73f6d68b37795b8: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
I20260928 11:02:47.824065 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.812751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.3.252.66:36993)
0928 11:02:47.812920 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.812927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.812950 (+ 23us) server_negotiation.cc:407] Connection header received
0928 11:02:47.813156 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.813161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.813265 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.813389 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:47.813815 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.814705 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.816462 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.816905 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.817007 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.822614 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:47.822623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:47.822638 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:47.823304 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:47.823360 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:47.823382 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:47.823461 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.823601 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.823719 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 11:02:47.824712 24282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:47.867570 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.867087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60198)
0928 11:02:47.867320 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.867411 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 11:02:47.868626 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.868295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60210)
0928 11:02:47.868432 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.868512 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 11:02:47.929045 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.928548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60222)
0928 11:02:47.928782 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.928875 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 11:02:47.930060 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.929702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60232)
0928 11:02:47.929876 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:47.929945 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 11:02:47.936198 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24160
I20260928 11:02:47.952334 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24290
I20260928 11:02:47.958967 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:47.953046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.3.252.67:43045)
0928 11:02:47.953244 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:47.953249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:47.953334 (+ 85us) server_negotiation.cc:407] Connection header received
0928 11:02:47.953528 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:47.953532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:47.953611 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:47.953696 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:47.954132 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.954874 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.956148 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:47.956533 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:47.956643 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:47.957600 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:47.957609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:47.957624 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:47.958288 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:47.958337 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:47.958357 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:47.958455 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:47.958580 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 11:02:47.958693 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 11:02:47.959604 24412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:48.011548 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:48.011016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60238)
0928 11:02:48.011285 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:48.011399 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 11:02:48.012653 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:48.012253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.65:43605 (local address 127.0.0.1:60248)
0928 11:02:48.012450 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 11:02:48.012525 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 11:02:48.048651 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24290
I20260928 11:02:48.062116 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 23939
I20260928 11:02:48.063192 24000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:48.174635 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 23939
2026-09-28T11:02:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2777 ms)
[----------] 6 tests from EncryptionPolicyTest (21036 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:48 dist-test-slave-djjw krb5kdc[24470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:48 dist-test-slave-djjw krb5kdc[24470](info): set up 2 sockets
Sep 28 11:02:48 dist-test-slave-djjw krb5kdc[24470](info): commencing operation
krb5kdc: starting...
W20260928 11:02:49.198366 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:49 dist-test-slave-djjw krb5kdc[24470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:02:49.451824 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:49.774468 24483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:49.774772 24483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:49.774871 24483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:49.782186 24483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:49.782325 24483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:49.782369 24483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:49.782434 24483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:49.782476 24483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:49.792883 24483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:49.794238 24483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:49.795526 24483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:49.806521 24488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:49.806499 24491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:49.806569 24489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:49.807520 24483 server_base.cc:1034] running on GCE node
I20260928 11:02:49.808205 24483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:49.809478 24483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:49.810675 24483 hybrid_clock.cc:648] HybridClock initialized: now 1790593369810638 us; error 27 us; skew 500 ppm
I20260928 11:02:49.811014 24483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:49 dist-test-slave-djjw krb5kdc[24470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:49.813992 24483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:02:49.815384 24483 webserver.cc:460] Webserver started at http://127.0.0.1:36245/ using document root <none> and password file <none>
I20260928 11:02:49.816066 24483 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:49.816210 24483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:49.816510 24483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:49.819494 24483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "84a97ee5a02f41959300cf9575b644a5"
format_stamp: "Formatted at 2026-09-28 11:02:49 on dist-test-slave-djjw"
server_key: "5a82275f533571df3ece1678ed3ea830"
server_key_iv: "c87f6e01d3d1225eee43084c50e5bc62"
server_key_version: "encryptionkey@0"
I20260928 11:02:49.820251 24483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "84a97ee5a02f41959300cf9575b644a5"
format_stamp: "Formatted at 2026-09-28 11:02:49 on dist-test-slave-djjw"
server_key: "5a82275f533571df3ece1678ed3ea830"
server_key_iv: "c87f6e01d3d1225eee43084c50e5bc62"
server_key_version: "encryptionkey@0"
I20260928 11:02:49.825394 24483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 11:02:49.828857 24498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:49.830178 24483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:02:49.830340 24483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "84a97ee5a02f41959300cf9575b644a5"
format_stamp: "Formatted at 2026-09-28 11:02:49 on dist-test-slave-djjw"
server_key: "5a82275f533571df3ece1678ed3ea830"
server_key_iv: "c87f6e01d3d1225eee43084c50e5bc62"
server_key_version: "encryptionkey@0"
I20260928 11:02:49.830518 24483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:49.848902 24483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:49.853106 24483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:49.853384 24483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:49.870927 24483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43023
I20260928 11:02:49.870950 24549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43023 every 8 connection(s)
I20260928 11:02:49.872586 24483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:49.875831 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24483
I20260928 11:02:49.876053 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:49.876667 24550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:49.876837 4081 external_mini_cluster.cc:1428] Setting key 70a80d75791f5bf514e43c52c714821a
I20260928 11:02:49.889243 24550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5: Bootstrap starting.
Sep 28 11:02:49 dist-test-slave-djjw krb5kdc[24470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 11:02:49.892536 24550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:49.893630 24550 log.cc:824] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:49.896453 24550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5: No bootstrap required, opened a new log
I20260928 11:02:49.898838 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:49.878840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41871 (local address 127.0.0.1:43023)
0928 11:02:49.879592 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:49.879607 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:49.879650 (+ 43us) server_negotiation.cc:407] Connection header received
0928 11:02:49.881402 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:49.881425 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:49.881867 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:49.882246 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:49.883664 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:49.884946 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:49.886286 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:49.887000 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:49.887175 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:49.891444 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:49.891452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:49.891467 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:49.891515 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:49.894617 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:49.895455 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:49.895462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:49.895468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:49.895638 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:49.896099 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:49.896103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:49.896105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:49.896648 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:49.896990 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:49.897468 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 11:02:49.897777 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":313,"threads_started":1}
I20260928 11:02:49.902779 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:02:49.904422 24550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a97ee5a02f41959300cf9575b644a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 11:02:49.904772 24550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:49.904875 24550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a97ee5a02f41959300cf9575b644a5, State: Initialized, Role: FOLLOWER
I20260928 11:02:49.905483 24550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a97ee5a02f41959300cf9575b644a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 11:02:49.905737 24550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:49.905931 24550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:49.906105 24550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:49.908870 24550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a97ee5a02f41959300cf9575b644a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 11:02:49.909373 24550 leader_election.cc:302] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84a97ee5a02f41959300cf9575b644a5; no voters:
I20260928 11:02:49.910025 24550 leader_election.cc:288] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:49.910125 24560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:49.912530 24550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:49.917037 24560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [term 1 LEADER]: Becoming Leader. State: Replica: 84a97ee5a02f41959300cf9575b644a5, State: Running, Role: LEADER
I20260928 11:02:49.917850 24560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a97ee5a02f41959300cf9575b644a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 11:02:49.920993 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84a97ee5a02f41959300cf9575b644a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a97ee5a02f41959300cf9575b644a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } } }
I20260928 11:02:49.921767 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:49.922230 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:49.907640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46603 (local address 127.0.0.1:43023)
0928 11:02:49.907856 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:49.907861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:49.908103 (+ 242us) server_negotiation.cc:407] Connection header received
0928 11:02:49.908279 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:49.908284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:49.908347 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:49.908426 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:49.909882 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:49.911005 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:49.912279 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:49.912824 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:49.912958 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:49.915418 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:49.915423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:49.915426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:49.915461 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:49.918460 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:49.919191 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:49.919196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:49.919198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:49.919314 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:49.919767 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:49.919773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:49.919776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:49.920210 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:49.920594 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:49.921648 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0928 11:02:49.921849 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 11:02:49.927340 24568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:49.927245 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84a97ee5a02f41959300cf9575b644a5. Latest consensus state: current_term: 1 leader_uuid: "84a97ee5a02f41959300cf9575b644a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a97ee5a02f41959300cf9575b644a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } } }
I20260928 11:02:49.927927 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:49.932763 24568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:49.942289 24568 catalog_manager.cc:1347] Generated new cluster ID: e5f39aa281ad4571ba63ef8581bcc8fb
I20260928 11:02:49.942438 24568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:49.957643 24568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:49.959106 24568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:49.978248 24568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84a97ee5a02f41959300cf9575b644a5: Generated new TSK 0
I20260928 11:02:49.979169 24568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:49.985060 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24483
I20260928 11:02:49.986932 24545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:50.079118 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24483
W20260928 11:02:50.093492 24468 connection.cc:535] client connection to 127.0.0.1:43023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43023): Connection reset by peer (error 104)
2026-09-28T11:02:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:50 dist-test-slave-djjw krb5kdc[24583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:50 dist-test-slave-djjw krb5kdc[24583](info): set up 2 sockets
Sep 28 11:02:50 dist-test-slave-djjw krb5kdc[24583](info): commencing operation
krb5kdc: starting...
W20260928 11:02:51.118321 4081 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:51 dist-test-slave-djjw krb5kdc[24583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:02:51.361388 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:51.685755 24596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:51.686069 24596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:51.686184 24596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:51.693647 24596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:51.693763 24596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:51.693819 24596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:51.693934 24596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:51.693993 24596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:51.703977 24596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:51.705338 24596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:51.706653 24596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:51.718029 24602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:51.718068 24601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:51.718155 24604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:51.719075 24596 server_base.cc:1034] running on GCE node
I20260928 11:02:51.719820 24596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:51.721340 24596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:51.722568 24596 hybrid_clock.cc:648] HybridClock initialized: now 1790593371722506 us; error 47 us; skew 500 ppm
I20260928 11:02:51.722945 24596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:51 dist-test-slave-djjw krb5kdc[24583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:51.726244 24596 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:02:51.727790 24596 webserver.cc:460] Webserver started at http://127.0.0.1:41343/ using document root <none> and password file <none>
I20260928 11:02:51.728477 24596 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:51.728587 24596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:51.728881 24596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:51.732017 24596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "6953e79e1b444c2c9dd77e1efff399f6"
format_stamp: "Formatted at 2026-09-28 11:02:51 on dist-test-slave-djjw"
server_key: "09197fb92829428b69a88c5835d93304"
server_key_iv: "29f8a6255f514b712bfc008a22a31397"
server_key_version: "encryptionkey@0"
I20260928 11:02:51.732766 24596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "6953e79e1b444c2c9dd77e1efff399f6"
format_stamp: "Formatted at 2026-09-28 11:02:51 on dist-test-slave-djjw"
server_key: "09197fb92829428b69a88c5835d93304"
server_key_iv: "29f8a6255f514b712bfc008a22a31397"
server_key_version: "encryptionkey@0"
I20260928 11:02:51.738349 24596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 11:02:51.741974 24611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:51.743420 24596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:51.743597 24596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "6953e79e1b444c2c9dd77e1efff399f6"
format_stamp: "Formatted at 2026-09-28 11:02:51 on dist-test-slave-djjw"
server_key: "09197fb92829428b69a88c5835d93304"
server_key_iv: "29f8a6255f514b712bfc008a22a31397"
server_key_version: "encryptionkey@0"
I20260928 11:02:51.743791 24596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:51.765172 24596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:51.769433 24596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:51.769699 24596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:51.787671 24596 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40689
I20260928 11:02:51.787700 24662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40689 every 8 connection(s)
I20260928 11:02:51.789058 24596 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:51.792905 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:51.795374 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24596
I20260928 11:02:51.795531 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:51.795996 4081 external_mini_cluster.cc:1428] Setting key 23335593020368a14382a6721ff3192e
I20260928 11:02:51.804440 24663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6: Bootstrap starting.
I20260928 11:02:51.807929 24663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:51.809024 24663 log.cc:824] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:51.809077 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:51.797797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44185 (local address 127.0.0.1:40689)
0928 11:02:51.798614 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:51.798625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:51.798666 (+ 41us) server_negotiation.cc:407] Connection header received
0928 11:02:51.800277 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:51.800300 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:51.800696 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:51.801016 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:51.801637 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:51.803031 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:51.804776 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:51.805452 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:51.805581 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:51.806854 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:51.806861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:51.806875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:51.807426 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:51.807502 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:51.807527 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:51.807646 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:51.807890 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 11:02:51.808072 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":334,"threads_started":1}
I20260928 11:02:51.811813 24663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6: No bootstrap required, opened a new log
I20260928 11:02:51.812798 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:02:51.819495 24663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6953e79e1b444c2c9dd77e1efff399f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40689 } }
I20260928 11:02:51.819903 24663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:51.819993 24663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6953e79e1b444c2c9dd77e1efff399f6, State: Initialized, Role: FOLLOWER
I20260928 11:02:51.820571 24663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6953e79e1b444c2c9dd77e1efff399f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40689 } }
I20260928 11:02:51.820802 24663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:51.820971 24663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:51.821135 24663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:51.822696 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:51.816907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34067 (local address 127.0.0.1:40689)
0928 11:02:51.817080 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:51.817086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:51.817404 (+ 318us) server_negotiation.cc:407] Connection header received
0928 11:02:51.817548 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:51.817552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:51.817616 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:51.817705 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:51.818203 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:51.818958 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:51.820303 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:51.820735 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:51.820851 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:51.821947 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:51.821951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:51.821954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:51.821978 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:51.822035 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:51.822052 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:51.822142 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:51.822316 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 11:02:51.822389 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 11:02:51.823906 24663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6953e79e1b444c2c9dd77e1efff399f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40689 } }
I20260928 11:02:51.824396 24663 leader_election.cc:302] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6953e79e1b444c2c9dd77e1efff399f6; no voters:
I20260928 11:02:51.824923 24663 leader_election.cc:288] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:51.825037 24673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:51.825637 24673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [term 1 LEADER]: Becoming Leader. State: Replica: 6953e79e1b444c2c9dd77e1efff399f6, State: Running, Role: LEADER
I20260928 11:02:51.826162 24673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6953e79e1b444c2c9dd77e1efff399f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40689 } }
I20260928 11:02:51.826850 24663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:51.828682 24674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6953e79e1b444c2c9dd77e1efff399f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6953e79e1b444c2c9dd77e1efff399f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40689 } } }
I20260928 11:02:51.828858 24675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6953e79e1b444c2c9dd77e1efff399f6. Latest consensus state: current_term: 1 leader_uuid: "6953e79e1b444c2c9dd77e1efff399f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6953e79e1b444c2c9dd77e1efff399f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40689 } } }
I20260928 11:02:51.829140 24674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:51.829195 24675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:51.830817 24678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:51.835772 24678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:51.845858 24678 catalog_manager.cc:1347] Generated new cluster ID: 0141e40dd2ad428aaa7ebef51e243056
I20260928 11:02:51.846005 24678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:51.865711 24678 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:51.867076 24678 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:51.877175 24678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6953e79e1b444c2c9dd77e1efff399f6: Generated new TSK 0
I20260928 11:02:51.878083 24678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:51.886514 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24596
W20260928 11:02:51.887687 24658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44185
W20260928 11:02:51.888402 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24596: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:02:51.888589 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24596
W20260928 11:02:51.902661 24581 connection.cc:535] client connection to 127.0.0.1:40689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40689): Connection reset by peer (error 104)
2026-09-28T11:02:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1807 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:51 dist-test-slave-djjw krb5kdc[24694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:51 dist-test-slave-djjw krb5kdc[24694](info): set up 2 sockets
Sep 28 11:02:51 dist-test-slave-djjw krb5kdc[24694](info): commencing operation
krb5kdc: starting...
W20260928 11:02:52.906199 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:53 dist-test-slave-djjw krb5kdc[24694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:02:53.154037 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:53.482332 24707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:53.482658 24707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:53.482796 24707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:53.489996 24707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:53.490130 24707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:53.490209 24707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:53.490288 24707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:53.490346 24707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:53.500216 24707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:53.501541 24707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:53.502851 24707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:53.514213 24715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:53.514261 24713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:53.514285 24712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:53.514984 24707 server_base.cc:1034] running on GCE node
I20260928 11:02:53.515880 24707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:53.517271 24707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:53.518507 24707 hybrid_clock.cc:648] HybridClock initialized: now 1790593373518461 us; error 35 us; skew 500 ppm
I20260928 11:02:53.518873 24707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:53 dist-test-slave-djjw krb5kdc[24694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:53.527446 24707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:02:53.529049 24707 webserver.cc:460] Webserver started at http://127.0.0.1:37057/ using document root <none> and password file <none>
I20260928 11:02:53.529733 24707 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:53.529865 24707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:53.530160 24707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:53.533218 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "79e94c9d265341ad816735be7bd58ad6"
format_stamp: "Formatted at 2026-09-28 11:02:53 on dist-test-slave-djjw"
server_key: "e668c4e8a008aff2db1375d74d523927"
server_key_iv: "82c549549c44a686a487ae3bcaeb9a7a"
server_key_version: "encryptionkey@0"
I20260928 11:02:53.534019 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "79e94c9d265341ad816735be7bd58ad6"
format_stamp: "Formatted at 2026-09-28 11:02:53 on dist-test-slave-djjw"
server_key: "e668c4e8a008aff2db1375d74d523927"
server_key_iv: "82c549549c44a686a487ae3bcaeb9a7a"
server_key_version: "encryptionkey@0"
I20260928 11:02:53.539132 24707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 11:02:53.542654 24722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:53.544028 24707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:53.544193 24707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "79e94c9d265341ad816735be7bd58ad6"
format_stamp: "Formatted at 2026-09-28 11:02:53 on dist-test-slave-djjw"
server_key: "e668c4e8a008aff2db1375d74d523927"
server_key_iv: "82c549549c44a686a487ae3bcaeb9a7a"
server_key_version: "encryptionkey@0"
I20260928 11:02:53.544394 24707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:53.550223 24707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:53.554286 24707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:53.554564 24707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:53.571398 24707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38799
I20260928 11:02:53.571434 24773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38799 every 8 connection(s)
I20260928 11:02:53.572786 24707 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:53.576295 24774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:53.577771 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24707
I20260928 11:02:53.577967 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:53.578469 4081 external_mini_cluster.cc:1428] Setting key cc42eec28a2285d8f1395ffd6778130d
I20260928 11:02:53.586055 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:53.580275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58775 (local address 127.0.0.1:38799)
0928 11:02:53.581034 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:53.581045 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:53.581091 (+ 46us) server_negotiation.cc:407] Connection header received
0928 11:02:53.582704 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:53.582727 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:53.583100 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:53.583448 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:53.584495 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:53.584502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:53.584515 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:53.584530 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:53.584588 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:53.584610 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:53.584769 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:53.584868 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 11:02:53.585042 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":336,"threads_started":1}
I20260928 11:02:53.589531 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:02:53.589713 24774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6: Bootstrap starting.
I20260928 11:02:53.593480 24774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:53.594760 24774 log.cc:824] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:53.596935 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:53.594624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57321 (local address 127.0.0.1:38799)
0928 11:02:53.594806 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:53.594812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:53.595073 (+ 261us) server_negotiation.cc:407] Connection header received
0928 11:02:53.595243 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:53.595248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:53.595338 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:53.595458 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:53.596362 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:53.596366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:53.596369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:53.596384 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:53.596420 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:53.596433 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:53.596517 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:53.596629 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 11:02:53.596673 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 11:02:53.597959 24774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6: No bootstrap required, opened a new log
I20260928 11:02:53.605962 24774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e94c9d265341ad816735be7bd58ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260928 11:02:53.606400 24774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:53.606500 24774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e94c9d265341ad816735be7bd58ad6, State: Initialized, Role: FOLLOWER
I20260928 11:02:53.607069 24774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e94c9d265341ad816735be7bd58ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260928 11:02:53.607366 24774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:53.607519 24774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:53.607692 24774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:53.609934 24774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e94c9d265341ad816735be7bd58ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260928 11:02:53.610410 24774 leader_election.cc:302] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79e94c9d265341ad816735be7bd58ad6; no voters:
I20260928 11:02:53.610947 24774 leader_election.cc:288] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:53.611075 24784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:53.611701 24784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 79e94c9d265341ad816735be7bd58ad6, State: Running, Role: LEADER
I20260928 11:02:53.612285 24784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e94c9d265341ad816735be7bd58ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260928 11:02:53.612901 24774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:53.615855 24786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79e94c9d265341ad816735be7bd58ad6. Latest consensus state: current_term: 1 leader_uuid: "79e94c9d265341ad816735be7bd58ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e94c9d265341ad816735be7bd58ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260928 11:02:53.616153 24786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:53.616619 24785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79e94c9d265341ad816735be7bd58ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e94c9d265341ad816735be7bd58ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260928 11:02:53.616974 24785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:53.618455 24791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:53.627225 24791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:53.636453 24791 catalog_manager.cc:1347] Generated new cluster ID: d69b538477d7483bb2ffae945c740ccb
I20260928 11:02:53.636585 24791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:53.660770 24791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:53.661208 24791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:53.667040 24791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79e94c9d265341ad816735be7bd58ad6: Generated new TSK 0
I20260928 11:02:53.667876 24791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:53.672925 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24707
W20260928 11:02:53.674078 24769 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58775
W20260928 11:02:53.674887 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24707: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:02:53.675257 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24707
2026-09-28T11:02:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1787 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:53 dist-test-slave-djjw krb5kdc[24805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:53 dist-test-slave-djjw krb5kdc[24805](info): set up 2 sockets
Sep 28 11:02:53 dist-test-slave-djjw krb5kdc[24805](info): commencing operation
krb5kdc: starting...
W20260928 11:02:54.690377 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:54 dist-test-slave-djjw krb5kdc[24805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:02:54.934487 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:55.258615 24818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:55.258949 24818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:55.259087 24818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:55.266600 24818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:55.266743 24818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:55.266820 24818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:55.266912 24818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:55.266973 24818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:55.277113 24818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:55.278492 24818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:55.279773 24818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:55.291172 24826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:55.291445 24823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:55.291206 24824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:55.292052 24818 server_base.cc:1034] running on GCE node
I20260928 11:02:55.292851 24818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:55.294446 24818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:55.295674 24818 hybrid_clock.cc:648] HybridClock initialized: now 1790593375295627 us; error 46 us; skew 500 ppm
I20260928 11:02:55.296036 24818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:55 dist-test-slave-djjw krb5kdc[24805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:55.299422 24818 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:02:55.301040 24818 webserver.cc:460] Webserver started at http://127.0.0.1:37431/ using document root <none> and password file <none>
I20260928 11:02:55.301751 24818 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:55.301914 24818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:55.302222 24818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:55.305294 24818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "415095f45ef1496fbffc4c638d2caa98"
format_stamp: "Formatted at 2026-09-28 11:02:55 on dist-test-slave-djjw"
server_key: "983eab3ed7ad71a23d120e475a159e49"
server_key_iv: "d06dc98aa78cebde1c041419a2c373fc"
server_key_version: "encryptionkey@0"
I20260928 11:02:55.306159 24818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "415095f45ef1496fbffc4c638d2caa98"
format_stamp: "Formatted at 2026-09-28 11:02:55 on dist-test-slave-djjw"
server_key: "983eab3ed7ad71a23d120e475a159e49"
server_key_iv: "d06dc98aa78cebde1c041419a2c373fc"
server_key_version: "encryptionkey@0"
I20260928 11:02:55.311626 24818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:02:55.315284 24833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:55.316660 24818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:02:55.316843 24818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "415095f45ef1496fbffc4c638d2caa98"
format_stamp: "Formatted at 2026-09-28 11:02:55 on dist-test-slave-djjw"
server_key: "983eab3ed7ad71a23d120e475a159e49"
server_key_iv: "d06dc98aa78cebde1c041419a2c373fc"
server_key_version: "encryptionkey@0"
I20260928 11:02:55.317039 24818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:55.342397 24818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:55.346644 24818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:55.346911 24818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:55.364135 24818 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35063
I20260928 11:02:55.364166 24884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35063 every 8 connection(s)
I20260928 11:02:55.365512 24818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:55.368434 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24818
I20260928 11:02:55.368619 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:55.369184 4081 external_mini_cluster.cc:1428] Setting key b2148114fd875b881738246d703fb463
I20260928 11:02:55.369817 24885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 11:02:55 dist-test-slave-djjw krb5kdc[24805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 11:02:55.384413 24885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98: Bootstrap starting.
I20260928 11:02:55.387764 24885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:55.389039 24885 log.cc:824] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:55.391865 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:55.370947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43173 (local address 127.0.0.1:35063)
0928 11:02:55.371706 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:55.371717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:55.371757 (+ 40us) server_negotiation.cc:407] Connection header received
0928 11:02:55.373271 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:55.373293 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:55.373699 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:55.374089 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:55.375872 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:55.377257 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:55.379017 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:55.379702 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:55.379833 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:55.384634 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:55.384645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:55.384661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:55.384703 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:55.387621 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:55.388550 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:55.388556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:55.388563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:55.388741 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:55.389152 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:55.389157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:55.389160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:55.389819 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:55.390211 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:55.390520 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 11:02:55.390890 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":336,"threads_started":1}
I20260928 11:02:55.392282 24885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98: No bootstrap required, opened a new log
I20260928 11:02:55.395375 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:02:55.400357 24885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415095f45ef1496fbffc4c638d2caa98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260928 11:02:55.400846 24885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:55.400955 24885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415095f45ef1496fbffc4c638d2caa98, State: Initialized, Role: FOLLOWER
I20260928 11:02:55.401671 24885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415095f45ef1496fbffc4c638d2caa98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260928 11:02:55.401970 24885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:55.402112 24885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:55.402297 24885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:55.405238 24885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415095f45ef1496fbffc4c638d2caa98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260928 11:02:55.405766 24885 leader_election.cc:302] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415095f45ef1496fbffc4c638d2caa98; no voters:
I20260928 11:02:55.406471 24885 leader_election.cc:288] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:55.406601 24895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:55.407826 24895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [term 1 LEADER]: Becoming Leader. State: Replica: 415095f45ef1496fbffc4c638d2caa98, State: Running, Role: LEADER
I20260928 11:02:55.408360 24895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415095f45ef1496fbffc4c638d2caa98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260928 11:02:55.408847 24885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:55.411967 24896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415095f45ef1496fbffc4c638d2caa98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415095f45ef1496fbffc4c638d2caa98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } } }
I20260928 11:02:55.412382 24896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:55.414664 24897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415095f45ef1496fbffc4c638d2caa98. Latest consensus state: current_term: 1 leader_uuid: "415095f45ef1496fbffc4c638d2caa98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415095f45ef1496fbffc4c638d2caa98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } } }
I20260928 11:02:55.415037 24897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:55.416146 24901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:55.417363 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:55.399695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42547 (local address 127.0.0.1:35063)
0928 11:02:55.399913 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:55.399919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:55.400250 (+ 331us) server_negotiation.cc:407] Connection header received
0928 11:02:55.400414 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:55.400419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:55.400488 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:55.400573 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:55.402610 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:55.403440 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:55.405202 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:55.405765 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:55.405945 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:55.408846 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:55.408853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:55.408858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:02:55.408897 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:55.412999 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:55.413987 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:55.413993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:55.413997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:55.414161 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:02:55.414676 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:02:55.414682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:02:55.414686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:02:55.415098 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:55.415430 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:55.416649 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0928 11:02:55.416914 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 11:02:55.421294 24901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:55.430868 24901 catalog_manager.cc:1347] Generated new cluster ID: 96fc3643fe874a44b5627496544d6561
I20260928 11:02:55.431006 24901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:55.447389 24901 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:55.448712 24901 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:55.456382 24901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 415095f45ef1496fbffc4c638d2caa98: Generated new TSK 0
I20260928 11:02:55.457310 24901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:55.465420 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24818
I20260928 11:02:55.466651 24880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:02:55.557925 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24818
2026-09-28T11:02:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:55 dist-test-slave-djjw krb5kdc[24917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:55 dist-test-slave-djjw krb5kdc[24917](info): set up 2 sockets
Sep 28 11:02:55 dist-test-slave-djjw krb5kdc[24917](info): commencing operation
krb5kdc: starting...
W20260928 11:02:56.582270 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:56 dist-test-slave-djjw krb5kdc[24917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:02:56.828482 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:57.159488 24930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:57.159793 24930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:57.159929 24930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:57.167233 24930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:57.167388 24930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:57.167471 24930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:57.167557 24930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:57.167613 24930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:57.177557 24930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:57.178972 24930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:57.180298 24930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:57.191831 24935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:57.191919 24936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:57.191937 24938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:57.192763 24930 server_base.cc:1034] running on GCE node
I20260928 11:02:57.193578 24930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:57.195070 24930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:57.196290 24930 hybrid_clock.cc:648] HybridClock initialized: now 1790593377196239 us; error 42 us; skew 500 ppm
I20260928 11:02:57.196667 24930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:57 dist-test-slave-djjw krb5kdc[24917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:57.199812 24930 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:02:57.201269 24930 webserver.cc:460] Webserver started at http://127.0.0.1:33405/ using document root <none> and password file <none>
I20260928 11:02:57.201987 24930 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:57.202105 24930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:57.202392 24930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:57.205470 24930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "d4c144a1baa049689ec08b443da1204f"
format_stamp: "Formatted at 2026-09-28 11:02:57 on dist-test-slave-djjw"
server_key: "dfdf2963e3b08785a7b39fff0b23c1f0"
server_key_iv: "2089df4d9c30b6b6b9fd3c6d839a33a5"
server_key_version: "encryptionkey@0"
I20260928 11:02:57.206285 24930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "d4c144a1baa049689ec08b443da1204f"
format_stamp: "Formatted at 2026-09-28 11:02:57 on dist-test-slave-djjw"
server_key: "dfdf2963e3b08785a7b39fff0b23c1f0"
server_key_iv: "2089df4d9c30b6b6b9fd3c6d839a33a5"
server_key_version: "encryptionkey@0"
I20260928 11:02:57.211793 24930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:02:57.215731 24945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:57.217207 24930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 11:02:57.217378 24930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "d4c144a1baa049689ec08b443da1204f"
format_stamp: "Formatted at 2026-09-28 11:02:57 on dist-test-slave-djjw"
server_key: "dfdf2963e3b08785a7b39fff0b23c1f0"
server_key_iv: "2089df4d9c30b6b6b9fd3c6d839a33a5"
server_key_version: "encryptionkey@0"
I20260928 11:02:57.217582 24930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:57.249938 24930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:57.254504 24930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:57.254783 24930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:57.272440 24930 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36047
I20260928 11:02:57.272464 24996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36047 every 8 connection(s)
I20260928 11:02:57.273880 24930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:57.277572 24997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:57.281783 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 24930
I20260928 11:02:57.281999 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:57.282423 4081 external_mini_cluster.cc:1428] Setting key f5f50349c99aadaf8d99b5d52109ebda
I20260928 11:02:57.288568 24997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f: Bootstrap starting.
I20260928 11:02:57.291702 24997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:57.292820 24997 log.cc:824] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:57.295851 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:57.284173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58305 (local address 127.0.0.1:36047)
0928 11:02:57.284971 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:57.284987 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:57.285027 (+ 40us) server_negotiation.cc:407] Connection header received
0928 11:02:57.287240 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:57.287265 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:57.287707 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:57.288059 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:57.288605 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:57.289958 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:57.291250 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:57.291977 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:57.292156 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:57.293379 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:57.293393 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:57.293414 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:57.294188 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:57.294284 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:57.294314 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:57.294426 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:57.294646 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 11:02:57.294915 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":354,"threads_started":1}
I20260928 11:02:57.295830 24997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f: No bootstrap required, opened a new log
I20260928 11:02:57.299656 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:02:57.303920 24997 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c144a1baa049689ec08b443da1204f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36047 } }
I20260928 11:02:57.304457 24997 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:57.304564 24997 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c144a1baa049689ec08b443da1204f, State: Initialized, Role: FOLLOWER
I20260928 11:02:57.305202 24997 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c144a1baa049689ec08b443da1204f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36047 } }
I20260928 11:02:57.305436 24997 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:57.305583 24997 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:57.305783 24997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:57.308527 24997 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c144a1baa049689ec08b443da1204f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36047 } }
I20260928 11:02:57.309170 24997 leader_election.cc:302] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4c144a1baa049689ec08b443da1204f; no voters:
I20260928 11:02:57.309844 24997 leader_election.cc:288] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:57.309984 25007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:57.310842 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:57.304182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59887 (local address 127.0.0.1:36047)
0928 11:02:57.304373 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:57.304379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:57.304653 (+ 274us) server_negotiation.cc:407] Connection header received
0928 11:02:57.304842 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:57.304846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:57.304914 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:57.304993 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:57.305532 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:57.306604 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:57.308012 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:02:57.308690 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:02:57.308811 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:02:57.309980 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:57.309986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:57.309991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:57.310013 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:57.310063 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:57.310085 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:57.310181 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:57.310363 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 11:02:57.310471 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 11:02:57.311203 25007 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [term 1 LEADER]: Becoming Leader. State: Replica: d4c144a1baa049689ec08b443da1204f, State: Running, Role: LEADER
I20260928 11:02:57.311863 25007 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c144a1baa049689ec08b443da1204f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36047 } }
I20260928 11:02:57.312194 24997 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:57.314898 25009 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4c144a1baa049689ec08b443da1204f. Latest consensus state: current_term: 1 leader_uuid: "d4c144a1baa049689ec08b443da1204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c144a1baa049689ec08b443da1204f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36047 } } }
I20260928 11:02:57.314987 25008 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4c144a1baa049689ec08b443da1204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c144a1baa049689ec08b443da1204f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36047 } } }
I20260928 11:02:57.315220 25009 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:57.315315 25008 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:57.316690 25016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:57.322438 25016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:57.332588 25016 catalog_manager.cc:1347] Generated new cluster ID: 8f9a05e5f1ef4a378d76232d55df1b4d
I20260928 11:02:57.332710 25016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:57.342667 25016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:57.344569 25016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:57.351661 25016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4c144a1baa049689ec08b443da1204f: Generated new TSK 0
I20260928 11:02:57.352702 25016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:57.359609 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24930
W20260928 11:02:57.360662 24992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58305
W20260928 11:02:57.361320 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 24930: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:02:57.361455 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 24930
2026-09-28T11:02:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1803 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:57 dist-test-slave-djjw krb5kdc[25028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:57 dist-test-slave-djjw krb5kdc[25028](info): set up 2 sockets
Sep 28 11:02:57 dist-test-slave-djjw krb5kdc[25028](info): commencing operation
krb5kdc: starting...
W20260928 11:02:58.378247 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:02:58 dist-test-slave-djjw krb5kdc[25028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:02:58.625204 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:02:58.951421 25041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:02:58.951716 25041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:02:58.951812 25041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:02:58.959014 25041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:02:58.959131 25041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:02:58.959175 25041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:02:58.959233 25041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:02:58.959276 25041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:02:58.969321 25041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:02:58.970607 25041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:02:58.971829 25041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:02:58.983537 25049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:58.983458 25046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:02:58.983606 25047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:02:58.984167 25041 server_base.cc:1034] running on GCE node
I20260928 11:02:58.984925 25041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:02:58.986436 25041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:02:58.987674 25041 hybrid_clock.cc:648] HybridClock initialized: now 1790593378987627 us; error 36 us; skew 500 ppm
I20260928 11:02:58.988060 25041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:02:58 dist-test-slave-djjw krb5kdc[25028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:02:58.994900 25041 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:02:58.996745 25041 webserver.cc:460] Webserver started at http://127.0.0.1:43637/ using document root <none> and password file <none>
I20260928 11:02:58.997488 25041 fs_manager.cc:359] Metadata directory not provided
I20260928 11:02:58.997629 25041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:02:58.997989 25041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:02:59.001148 25041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "69c827c7c2bd43f598224d39525520c0"
format_stamp: "Formatted at 2026-09-28 11:02:58 on dist-test-slave-djjw"
server_key: "3eda0ec456f57464bcfd183440423647"
server_key_iv: "2540f81ef46d720959fed9fdaf526f86"
server_key_version: "encryptionkey@0"
I20260928 11:02:59.002043 25041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "69c827c7c2bd43f598224d39525520c0"
format_stamp: "Formatted at 2026-09-28 11:02:58 on dist-test-slave-djjw"
server_key: "3eda0ec456f57464bcfd183440423647"
server_key_iv: "2540f81ef46d720959fed9fdaf526f86"
server_key_version: "encryptionkey@0"
I20260928 11:02:59.007725 25041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 11:02:59.011881 25056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:02:59.013387 25041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 11:02:59.013615 25041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "69c827c7c2bd43f598224d39525520c0"
format_stamp: "Formatted at 2026-09-28 11:02:58 on dist-test-slave-djjw"
server_key: "3eda0ec456f57464bcfd183440423647"
server_key_iv: "2540f81ef46d720959fed9fdaf526f86"
server_key_version: "encryptionkey@0"
I20260928 11:02:59.013819 25041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:02:59.019619 25041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:02:59.023546 25041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:02:59.023819 25041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:02:59.041046 25041 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45377
I20260928 11:02:59.041109 25107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45377 every 8 connection(s)
I20260928 11:02:59.042495 25041 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:02:59.046360 25108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:02:59.048842 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25041
I20260928 11:02:59.049026 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:02:59.049558 4081 external_mini_cluster.cc:1428] Setting key 14f024ee7cdf5e4e96d7321e6a681c6d
I20260928 11:02:59.057773 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:59.051656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49045 (local address 127.0.0.1:45377)
0928 11:02:59.052472 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:59.052484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:59.052531 (+ 47us) server_negotiation.cc:407] Connection header received
0928 11:02:59.054280 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:59.054306 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:59.054683 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:59.055075 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:59.056124 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:59.056131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:59.056145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:59.056161 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:59.056229 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:59.056253 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:59.056481 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:59.056638 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 11:02:59.056826 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":355,"threads_started":1}
I20260928 11:02:59.058625 25108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0: Bootstrap starting.
I20260928 11:02:59.061357 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:02:59.061898 25108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0: Neither blocks nor log segments found. Creating new log.
I20260928 11:02:59.062922 25108 log.cc:824] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0: Log is configured to *not* fsync() on all Append() calls
I20260928 11:02:59.066313 25108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0: No bootstrap required, opened a new log
I20260928 11:02:59.068029 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:02:59.065646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40825 (local address 127.0.0.1:45377)
0928 11:02:59.065789 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 11:02:59.065793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:02:59.066313 (+ 520us) server_negotiation.cc:407] Connection header received
0928 11:02:59.066390 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:02:59.066395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:02:59.066454 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:02:59.066558 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:02:59.067420 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:02:59.067426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:02:59.067431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:02:59.067451 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:02:59.067497 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:02:59.067517 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:02:59.067658 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 11:02:59.067702 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 11:02:59.067758 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 11:02:59.074579 25108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c827c7c2bd43f598224d39525520c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260928 11:02:59.074960 25108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:02:59.075033 25108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c827c7c2bd43f598224d39525520c0, State: Initialized, Role: FOLLOWER
I20260928 11:02:59.075569 25108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c827c7c2bd43f598224d39525520c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260928 11:02:59.075795 25108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:02:59.075896 25108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:02:59.076020 25108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:02:59.078586 25108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c827c7c2bd43f598224d39525520c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260928 11:02:59.079083 25108 leader_election.cc:302] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c827c7c2bd43f598224d39525520c0; no voters:
I20260928 11:02:59.079586 25108 leader_election.cc:288] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:02:59.079737 25118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:02:59.080358 25118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [term 1 LEADER]: Becoming Leader. State: Replica: 69c827c7c2bd43f598224d39525520c0, State: Running, Role: LEADER
I20260928 11:02:59.080878 25118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c827c7c2bd43f598224d39525520c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260928 11:02:59.081502 25108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:02:59.083441 25119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c827c7c2bd43f598224d39525520c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c827c7c2bd43f598224d39525520c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } } }
I20260928 11:02:59.083901 25119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:59.083459 25120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c827c7c2bd43f598224d39525520c0. Latest consensus state: current_term: 1 leader_uuid: "69c827c7c2bd43f598224d39525520c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c827c7c2bd43f598224d39525520c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } } }
I20260928 11:02:59.084342 25120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0 [sys.catalog]: This master's current role is: LEADER
I20260928 11:02:59.087224 25125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:02:59.092538 25125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:02:59.102474 25125 catalog_manager.cc:1347] Generated new cluster ID: 918a504551e049ec9444619c3b650105
I20260928 11:02:59.102669 25125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:02:59.136130 25125 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:02:59.136535 25125 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:02:59.142030 25125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c827c7c2bd43f598224d39525520c0: Generated new TSK 0
I20260928 11:02:59.142860 25125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:02:59.148659 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25041
W20260928 11:02:59.149668 25103 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49045
W20260928 11:02:59.150331 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25041: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:02:59.150456 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25041
2026-09-28T11:02:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1788 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:02:59 dist-test-slave-djjw krb5kdc[25139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:02:59 dist-test-slave-djjw krb5kdc[25139](info): set up 2 sockets
Sep 28 11:02:59 dist-test-slave-djjw krb5kdc[25139](info): commencing operation
krb5kdc: starting...
W20260928 11:03:00.162228 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:00 dist-test-slave-djjw krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:03:00.416981 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:00.743270 25152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:00.743561 25152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:00.743664 25152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:00.751112 25152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:00.751225 25152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:00.751273 25152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:00.751331 25152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:00.751376 25152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:00.761152 25152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:00.762427 25152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:00.763691 25152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:00.774989 25157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:00.775076 25160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:00.775031 25158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:00.775770 25152 server_base.cc:1034] running on GCE node
I20260928 11:03:00.776592 25152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:00.778095 25152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:00.779321 25152 hybrid_clock.cc:648] HybridClock initialized: now 1790593380779266 us; error 48 us; skew 500 ppm
I20260928 11:03:00.779672 25152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:00 dist-test-slave-djjw krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:00.782909 25152 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:03:00.784423 25152 webserver.cc:460] Webserver started at http://127.0.0.1:33297/ using document root <none> and password file <none>
I20260928 11:03:00.785140 25152 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:00.785261 25152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:00.785552 25152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:00.788678 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "f7a0dfec4a344b11b5b7b27531a94c88"
format_stamp: "Formatted at 2026-09-28 11:03:00 on dist-test-slave-djjw"
server_key: "08d1fe1c4dd4416ec932ea4a3ce0eebe"
server_key_iv: "4f0a2159acb292b7066606d86590955e"
server_key_version: "encryptionkey@0"
I20260928 11:03:00.789469 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "f7a0dfec4a344b11b5b7b27531a94c88"
format_stamp: "Formatted at 2026-09-28 11:03:00 on dist-test-slave-djjw"
server_key: "08d1fe1c4dd4416ec932ea4a3ce0eebe"
server_key_iv: "4f0a2159acb292b7066606d86590955e"
server_key_version: "encryptionkey@0"
I20260928 11:03:00.794899 25152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 11:03:00.798436 25167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:00.799825 25152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:03:00.800001 25152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "f7a0dfec4a344b11b5b7b27531a94c88"
format_stamp: "Formatted at 2026-09-28 11:03:00 on dist-test-slave-djjw"
server_key: "08d1fe1c4dd4416ec932ea4a3ce0eebe"
server_key_iv: "4f0a2159acb292b7066606d86590955e"
server_key_version: "encryptionkey@0"
I20260928 11:03:00.800199 25152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:00.819906 25152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:00.824324 25152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:00.824611 25152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:00.841409 25152 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40191
I20260928 11:03:00.841449 25218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40191 every 8 connection(s)
I20260928 11:03:00.842828 25152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:00.846508 25219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:00.851488 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25152
I20260928 11:03:00.851681 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:00.852097 4081 external_mini_cluster.cc:1428] Setting key 22fbd43667fe6b44e318c06016cac494
I20260928 11:03:00.858659 25219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88: Bootstrap starting.
I20260928 11:03:00.862303 25219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:00.863382 25219 log.cc:824] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:00.866400 25219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88: No bootstrap required, opened a new log
Sep 28 11:03:00 dist-test-slave-djjw krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 11:03:00.876529 25219 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260928 11:03:00.876995 25219 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:00.877112 25219 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a0dfec4a344b11b5b7b27531a94c88, State: Initialized, Role: FOLLOWER
I20260928 11:03:00.877179 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:00.854007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:58881 (local address 127.0.0.1:40191)
0928 11:03:00.854821 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:00.854840 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:00.854894 (+ 54us) server_negotiation.cc:407] Connection header received
0928 11:03:00.857300 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:00.857328 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:00.857869 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:00.858314 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:00.859663 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:00.861591 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:00.862983 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:00.863996 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:00.864181 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:00.868503 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:00.868515 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:00.868535 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:00.868586 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:00.872424 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:00.873273 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:00.873282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:00.873291 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:00.873502 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:00.873934 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:00.873942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:00.873946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:00.874724 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:00.875057 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:00.875443 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 11:03:00.875804 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":307,"threads_started":1}
I20260928 11:03:00.877864 25219 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260928 11:03:00.878098 25219 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:00.878228 25219 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:00.878382 25219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:00.881229 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:00.881295 25219 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260928 11:03:00.881865 25219 leader_election.cc:302] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7a0dfec4a344b11b5b7b27531a94c88; no voters:
I20260928 11:03:00.882486 25219 leader_election.cc:288] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:00.882588 25224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:00.883702 25224 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [term 1 LEADER]: Becoming Leader. State: Replica: f7a0dfec4a344b11b5b7b27531a94c88, State: Running, Role: LEADER
I20260928 11:03:00.884208 25224 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260928 11:03:00.884722 25219 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:00.887032 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7a0dfec4a344b11b5b7b27531a94c88. Latest consensus state: current_term: 1 leader_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } } }
I20260928 11:03:00.887413 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:00.887313 25225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0dfec4a344b11b5b7b27531a94c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } } }
I20260928 11:03:00.887585 25225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:00.888315 25237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:00.901129 25237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:00.914258 25237 catalog_manager.cc:1347] Generated new cluster ID: 70ae0dfb6aee4da1b05b88e9913d48bf
I20260928 11:03:00.914379 25237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:00.915094 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:00.887089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:58177 (local address 127.0.0.1:40191)
0928 11:03:00.887315 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:00.887322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:00.896855 (+ 9533us) server_negotiation.cc:407] Connection header received
0928 11:03:00.897036 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:00.897042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:00.897145 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:00.897249 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:00.899468 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:00.900785 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:00.902611 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:00.903318 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:00.903424 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:00.907026 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:00.907034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:00.907039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:00.907082 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:00.910873 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:00.911963 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:00.911969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:00.911973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:00.912140 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:00.912739 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:00.912744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:00.912748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:00.913235 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:00.914472 (+ 1237us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:00.914684 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 11:03:00.914839 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 11:03:00.948513 25237 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:00.949846 25237 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:00.963727 25237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7a0dfec4a344b11b5b7b27531a94c88: Generated new TSK 0
I20260928 11:03:00.964805 25237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:00.970382 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25152
I20260928 11:03:00.971755 25214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:01.064512 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25152
W20260928 11:03:01.079443 25137 connection.cc:535] client connection to 127.0.0.1:40191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40191): Connection reset by peer (error 104)
2026-09-28T11:03:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:01 dist-test-slave-djjw krb5kdc[25251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:01 dist-test-slave-djjw krb5kdc[25251](info): set up 2 sockets
Sep 28 11:03:01 dist-test-slave-djjw krb5kdc[25251](info): commencing operation
krb5kdc: starting...
W20260928 11:03:02.082276 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:02 dist-test-slave-djjw krb5kdc[25251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:03:02.331148 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:02.657555 25264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:02.657903 25264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:02.658046 25264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:02.665548 25264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:02.665678 25264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:02.665751 25264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:02.665858 25264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:02.665921 25264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:02.675981 25264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:02.677311 25264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:02.678642 25264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:02.690344 25269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:02.690357 25272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:02.690361 25270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:02.691817 25264 server_base.cc:1034] running on GCE node
I20260928 11:03:02.692616 25264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:02.694072 25264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:02.695281 25264 hybrid_clock.cc:648] HybridClock initialized: now 1790593382695231 us; error 39 us; skew 500 ppm
I20260928 11:03:02.695613 25264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:02 dist-test-slave-djjw krb5kdc[25251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:02.698596 25264 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:03:02.700007 25264 webserver.cc:460] Webserver started at http://127.0.0.1:43513/ using document root <none> and password file <none>
I20260928 11:03:02.700642 25264 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:02.700735 25264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:02.700977 25264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:02.703950 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "598241d9a65745568957148ed1e5a6ad"
format_stamp: "Formatted at 2026-09-28 11:03:02 on dist-test-slave-djjw"
server_key: "ac2d266f77693e3238f12af0e2ac6447"
server_key_iv: "154622ab6d8c96752019c821dab98247"
server_key_version: "encryptionkey@0"
I20260928 11:03:02.704674 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "598241d9a65745568957148ed1e5a6ad"
format_stamp: "Formatted at 2026-09-28 11:03:02 on dist-test-slave-djjw"
server_key: "ac2d266f77693e3238f12af0e2ac6447"
server_key_iv: "154622ab6d8c96752019c821dab98247"
server_key_version: "encryptionkey@0"
I20260928 11:03:02.709784 25264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 11:03:02.713581 25279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:02.715044 25264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:03:02.715240 25264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "598241d9a65745568957148ed1e5a6ad"
format_stamp: "Formatted at 2026-09-28 11:03:02 on dist-test-slave-djjw"
server_key: "ac2d266f77693e3238f12af0e2ac6447"
server_key_iv: "154622ab6d8c96752019c821dab98247"
server_key_version: "encryptionkey@0"
I20260928 11:03:02.715431 25264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:02.742972 25264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:02.747207 25264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:02.747484 25264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:02.765247 25264 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35153
I20260928 11:03:02.765269 25330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35153 every 8 connection(s)
I20260928 11:03:02.766666 25264 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:02.770314 25331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:02.775202 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25264
I20260928 11:03:02.775393 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:02.775938 4081 external_mini_cluster.cc:1428] Setting key 86070c455d43141812db00dac8864e6d
I20260928 11:03:02.781963 25331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad: Bootstrap starting.
I20260928 11:03:02.785601 25331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:02.786674 25331 log.cc:824] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:02.789515 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:02.777778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:33059 (local address 127.0.0.1:35153)
0928 11:03:02.778550 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:02.778562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:02.778607 (+ 45us) server_negotiation.cc:407] Connection header received
0928 11:03:02.780436 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:02.780459 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:02.780851 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:02.781167 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:02.781716 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:02.783142 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:02.784978 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:02.785675 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:02.785917 (+ 242us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:02.787099 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:02.787116 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:02.787136 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:02.787869 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:02.787960 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:02.787993 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:02.788160 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:02.788397 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 11:03:02.788578 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":326,"threads_started":1}
I20260928 11:03:02.789813 25331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad: No bootstrap required, opened a new log
I20260928 11:03:02.793202 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:02.798080 25331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598241d9a65745568957148ed1e5a6ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35153 } }
I20260928 11:03:02.798512 25331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:02.798609 25331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598241d9a65745568957148ed1e5a6ad, State: Initialized, Role: FOLLOWER
I20260928 11:03:02.799203 25331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598241d9a65745568957148ed1e5a6ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35153 } }
I20260928 11:03:02.799434 25331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:02.799576 25331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:02.799732 25331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:02.802115 25331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598241d9a65745568957148ed1e5a6ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35153 } }
I20260928 11:03:02.802613 25331 leader_election.cc:302] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 598241d9a65745568957148ed1e5a6ad; no voters:
I20260928 11:03:02.803201 25331 leader_election.cc:288] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:02.803292 25341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:02.803823 25341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [term 1 LEADER]: Becoming Leader. State: Replica: 598241d9a65745568957148ed1e5a6ad, State: Running, Role: LEADER
I20260928 11:03:02.804651 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:02.797449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:37447 (local address 127.0.0.1:35153)
0928 11:03:02.797674 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:02.797681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:02.798009 (+ 328us) server_negotiation.cc:407] Connection header received
0928 11:03:02.798211 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:02.798214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:02.798279 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:02.798373 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:02.798884 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:02.799697 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:02.800995 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:02.801453 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:02.801565 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:02.803624 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:02.803631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:02.803635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:02.803660 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:02.803729 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:02.803752 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:02.803887 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:02.804269 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 11:03:02.804321 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 11:03:02.804658 25341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598241d9a65745568957148ed1e5a6ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35153 } }
I20260928 11:03:02.805765 25331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:02.808662 25343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 598241d9a65745568957148ed1e5a6ad. Latest consensus state: current_term: 1 leader_uuid: "598241d9a65745568957148ed1e5a6ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598241d9a65745568957148ed1e5a6ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35153 } } }
I20260928 11:03:02.809288 25342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "598241d9a65745568957148ed1e5a6ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598241d9a65745568957148ed1e5a6ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35153 } } }
I20260928 11:03:02.809582 25343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:02.809710 25342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:02.811899 25347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:02.817279 25347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:02.836331 25347 catalog_manager.cc:1347] Generated new cluster ID: 2e0da61ca5aa49e0b138ecbc23dba63a
I20260928 11:03:02.836473 25347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:02.849565 25347 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:02.850994 25347 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:02.862424 25347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 598241d9a65745568957148ed1e5a6ad: Generated new TSK 0
I20260928 11:03:02.863291 25347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:02.871400 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25264
W20260928 11:03:02.872824 25326 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:33059
W20260928 11:03:02.873585 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25264: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:02.873720 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25264
W20260928 11:03:02.887815 25249 connection.cc:535] client connection to 127.0.0.1:35153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35153): Connection reset by peer (error 104)
2026-09-28T11:03:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:02 dist-test-slave-djjw krb5kdc[25362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:02 dist-test-slave-djjw krb5kdc[25362](info): set up 2 sockets
Sep 28 11:03:02 dist-test-slave-djjw krb5kdc[25362](info): commencing operation
krb5kdc: starting...
W20260928 11:03:03.890348 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:04 dist-test-slave-djjw krb5kdc[25362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:03:04.141750 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:04.470312 25375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:04.470619 25375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:04.470754 25375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:04.478065 25375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:04.478188 25375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:04.478261 25375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:04.478338 25375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:04.478389 25375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:04.488515 25375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:04.489869 25375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:04.491143 25375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:04.502853 25381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:04.502905 25380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:04.502947 25383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:04.503886 25375 server_base.cc:1034] running on GCE node
I20260928 11:03:04.504688 25375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:04.506222 25375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:04.507426 25375 hybrid_clock.cc:648] HybridClock initialized: now 1790593384507383 us; error 43 us; skew 500 ppm
I20260928 11:03:04.507779 25375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:04 dist-test-slave-djjw krb5kdc[25362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:04.514441 25375 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:03:04.516155 25375 webserver.cc:460] Webserver started at http://127.0.0.1:45761/ using document root <none> and password file <none>
I20260928 11:03:04.516887 25375 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:04.517010 25375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:04.517316 25375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:04.520468 25375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "4a19edf2ecc048ed8ff051a4671b8bf7"
format_stamp: "Formatted at 2026-09-28 11:03:04 on dist-test-slave-djjw"
server_key: "e654515d7f5f72adab75f82a35a146bd"
server_key_iv: "b95ddfa051e6d2e9416004f82af72b10"
server_key_version: "encryptionkey@0"
I20260928 11:03:04.521301 25375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "4a19edf2ecc048ed8ff051a4671b8bf7"
format_stamp: "Formatted at 2026-09-28 11:03:04 on dist-test-slave-djjw"
server_key: "e654515d7f5f72adab75f82a35a146bd"
server_key_iv: "b95ddfa051e6d2e9416004f82af72b10"
server_key_version: "encryptionkey@0"
I20260928 11:03:04.526762 25375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 11:03:04.530355 25390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:04.531848 25375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 11:03:04.532027 25375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "4a19edf2ecc048ed8ff051a4671b8bf7"
format_stamp: "Formatted at 2026-09-28 11:03:04 on dist-test-slave-djjw"
server_key: "e654515d7f5f72adab75f82a35a146bd"
server_key_iv: "b95ddfa051e6d2e9416004f82af72b10"
server_key_version: "encryptionkey@0"
I20260928 11:03:04.532241 25375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:04.538246 25375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:04.542783 25375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:04.543077 25375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:04.560581 25375 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44961
I20260928 11:03:04.560619 25441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44961 every 8 connection(s)
I20260928 11:03:04.562067 25375 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:04.565891 25442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:04.568034 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25375
I20260928 11:03:04.568193 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:04.568610 4081 external_mini_cluster.cc:1428] Setting key cc7e7b7755755887815fd2001f8b6c97
I20260928 11:03:04.576638 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:04.570470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:53703 (local address 127.0.0.1:44961)
0928 11:03:04.571247 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:04.571258 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:04.571301 (+ 43us) server_negotiation.cc:407] Connection header received
0928 11:03:04.572939 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:04.572962 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:04.573314 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:04.573653 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:04.574760 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:04.574770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:04.574783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:04.574800 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:04.574869 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:04.574892 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:04.575117 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:04.575266 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 11:03:04.575453 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":318,"threads_started":1}
I20260928 11:03:04.578253 25442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7: Bootstrap starting.
I20260928 11:03:04.580765 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:04.581605 25442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:04.582845 25442 log.cc:824] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:04.586018 25442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7: No bootstrap required, opened a new log
I20260928 11:03:04.587855 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:04.585138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:59661 (local address 127.0.0.1:44961)
0928 11:03:04.585364 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:04.585370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:04.585762 (+ 392us) server_negotiation.cc:407] Connection header received
0928 11:03:04.585964 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:04.585968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:04.586023 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:04.586106 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:04.587137 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:04.587141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:04.587145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:04.587161 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:04.587204 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:04.587236 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:04.587346 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:04.587460 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 11:03:04.587523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 11:03:04.594024 25442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260928 11:03:04.594460 25442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:04.594550 25442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a19edf2ecc048ed8ff051a4671b8bf7, State: Initialized, Role: FOLLOWER
I20260928 11:03:04.595165 25442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260928 11:03:04.595396 25442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:04.595525 25442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:04.595682 25442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:04.597959 25442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260928 11:03:04.598488 25442 leader_election.cc:302] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a19edf2ecc048ed8ff051a4671b8bf7; no voters:
I20260928 11:03:04.599043 25442 leader_election.cc:288] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:04.599179 25452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:04.599846 25452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [term 1 LEADER]: Becoming Leader. State: Replica: 4a19edf2ecc048ed8ff051a4671b8bf7, State: Running, Role: LEADER
I20260928 11:03:04.600401 25452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260928 11:03:04.601002 25442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:04.603238 25454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a19edf2ecc048ed8ff051a4671b8bf7. Latest consensus state: current_term: 1 leader_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } } }
I20260928 11:03:04.603260 25453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a19edf2ecc048ed8ff051a4671b8bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } } }
I20260928 11:03:04.603735 25454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:04.603735 25453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:04.604511 25459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:04.610273 25459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:04.624799 25459 catalog_manager.cc:1347] Generated new cluster ID: b6765391386d4ae28bb444b7a3703587
I20260928 11:03:04.624933 25459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:04.645045 25459 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:04.645591 25459 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:04.662482 25459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a19edf2ecc048ed8ff051a4671b8bf7: Generated new TSK 0
I20260928 11:03:04.663369 25459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:04.674084 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25375
W20260928 11:03:04.675209 25437 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:53703
W20260928 11:03:04.675976 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25375: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:04.676105 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25375
2026-09-28T11:03:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1802 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:04 dist-test-slave-djjw krb5kdc[25473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:04 dist-test-slave-djjw krb5kdc[25473](info): set up 2 sockets
Sep 28 11:03:04 dist-test-slave-djjw krb5kdc[25473](info): commencing operation
krb5kdc: starting...
W20260928 11:03:05.706310 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:05 dist-test-slave-djjw krb5kdc[25473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:03:05.957942 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:06.286931 25486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:06.287281 25486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:06.287420 25486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:06.295140 25486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:06.295293 25486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:06.295374 25486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:06.295454 25486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:06.295511 25486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:06.305913 25486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:06.307322 25486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:06.308676 25486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:06.321082 25491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:06.321000 25492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:06.321036 25494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:06.322178 25486 server_base.cc:1034] running on GCE node
I20260928 11:03:06.323064 25486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:06.324590 25486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:06.325913 25486 hybrid_clock.cc:648] HybridClock initialized: now 1790593386325850 us; error 52 us; skew 500 ppm
I20260928 11:03:06.326305 25486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:06 dist-test-slave-djjw krb5kdc[25473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:06.329546 25486 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:03:06.331331 25486 webserver.cc:460] Webserver started at http://127.0.0.1:39607/ using document root <none> and password file <none>
I20260928 11:03:06.332057 25486 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:06.332185 25486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:06.332472 25486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:06.335580 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "ee00b1b37e094a2ba945d3466fa71b32"
format_stamp: "Formatted at 2026-09-28 11:03:06 on dist-test-slave-djjw"
server_key: "b0f5ff04a2d873772596bd81cf09a9bb"
server_key_iv: "addc00bae1e51dd5ebcf295300c66a3d"
server_key_version: "encryptionkey@0"
I20260928 11:03:06.336395 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "ee00b1b37e094a2ba945d3466fa71b32"
format_stamp: "Formatted at 2026-09-28 11:03:06 on dist-test-slave-djjw"
server_key: "b0f5ff04a2d873772596bd81cf09a9bb"
server_key_iv: "addc00bae1e51dd5ebcf295300c66a3d"
server_key_version: "encryptionkey@0"
I20260928 11:03:06.342340 25486 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 11:03:06.346079 25501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:06.347566 25486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:03:06.347761 25486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "ee00b1b37e094a2ba945d3466fa71b32"
format_stamp: "Formatted at 2026-09-28 11:03:06 on dist-test-slave-djjw"
server_key: "b0f5ff04a2d873772596bd81cf09a9bb"
server_key_iv: "addc00bae1e51dd5ebcf295300c66a3d"
server_key_version: "encryptionkey@0"
I20260928 11:03:06.347965 25486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:06.363515 25486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:06.367941 25486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:06.368219 25486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:06.386442 25486 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33733
I20260928 11:03:06.386474 25552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33733 every 8 connection(s)
I20260928 11:03:06.387890 25486 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:06.391773 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:06.392338 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25486
I20260928 11:03:06.392496 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:06.392925 4081 external_mini_cluster.cc:1428] Setting key 9adfd52e88f2595d0fbc97abe5238391
I20260928 11:03:06.404062 25553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32: Bootstrap starting.
Sep 28 11:03:06 dist-test-slave-djjw krb5kdc[25473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 11:03:06.407611 25553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:06.408654 25553 log.cc:824] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:06.411580 25553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32: No bootstrap required, opened a new log
I20260928 11:03:06.415753 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:06.394671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:49727 (local address 127.0.0.1:33733)
0928 11:03:06.395460 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:06.395472 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:06.395513 (+ 41us) server_negotiation.cc:407] Connection header received
0928 11:03:06.397876 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:06.397907 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:06.398360 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:06.398736 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:06.400188 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:06.401514 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:06.402891 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:06.403585 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:06.403745 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:06.408091 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:06.408100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:06.408115 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:06.408155 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:06.411268 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:06.412163 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:06.412170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:06.412177 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:06.412351 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:06.412800 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:06.412804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:06.412807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:06.413363 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:06.413700 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:06.414298 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 11:03:06.414640 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":335,"threads_started":1}
I20260928 11:03:06.419893 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:06.419811 25553 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee00b1b37e094a2ba945d3466fa71b32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33733 } }
I20260928 11:03:06.420224 25553 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:06.420311 25553 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee00b1b37e094a2ba945d3466fa71b32, State: Initialized, Role: FOLLOWER
I20260928 11:03:06.420849 25553 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee00b1b37e094a2ba945d3466fa71b32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33733 } }
I20260928 11:03:06.421077 25553 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:06.421201 25553 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:06.421361 25553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:06.423854 25553 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee00b1b37e094a2ba945d3466fa71b32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33733 } }
I20260928 11:03:06.424504 25553 leader_election.cc:302] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee00b1b37e094a2ba945d3466fa71b32; no voters:
I20260928 11:03:06.425261 25553 leader_election.cc:288] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:06.425393 25563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:06.428833 25563 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [term 1 LEADER]: Becoming Leader. State: Replica: ee00b1b37e094a2ba945d3466fa71b32, State: Running, Role: LEADER
I20260928 11:03:06.429353 25553 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:06.429468 25563 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee00b1b37e094a2ba945d3466fa71b32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33733 } }
I20260928 11:03:06.432669 25564 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee00b1b37e094a2ba945d3466fa71b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee00b1b37e094a2ba945d3466fa71b32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33733 } } }
I20260928 11:03:06.433084 25564 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:06.436578 25571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:06.436169 25565 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee00b1b37e094a2ba945d3466fa71b32. Latest consensus state: current_term: 1 leader_uuid: "ee00b1b37e094a2ba945d3466fa71b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee00b1b37e094a2ba945d3466fa71b32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33733 } } }
I20260928 11:03:06.436846 25565 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:06.442735 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:06.424533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:43409 (local address 127.0.0.1:33733)
0928 11:03:06.424707 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:06.424713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:06.425230 (+ 517us) server_negotiation.cc:407] Connection header received
0928 11:03:06.425482 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:06.425491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:06.425582 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:06.425692 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:06.427390 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:06.428632 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:06.430108 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:06.430614 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:06.430770 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:06.434135 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:06.434142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:06.434147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:06.434188 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:06.438374 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:06.439570 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:06.439576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:06.439579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:06.439751 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:06.440353 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:06.440358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:06.440361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:06.440814 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:06.441094 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:06.442120 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0928 11:03:06.442323 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 11:03:06.442744 25571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:06.456219 25571 catalog_manager.cc:1347] Generated new cluster ID: 4e490261f38f4111aa241f2d2d002295
I20260928 11:03:06.456367 25571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:06.475909 25571 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:06.477317 25571 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:06.487913 25571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee00b1b37e094a2ba945d3466fa71b32: Generated new TSK 0
I20260928 11:03:06.488831 25571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:06.498559 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25486
I20260928 11:03:06.499987 25548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:06.593179 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25486
W20260928 11:03:06.607949 25471 connection.cc:535] client connection to 127.0.0.1:33733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33733): Connection reset by peer (error 104)
2026-09-28T11:03:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:06 dist-test-slave-djjw krb5kdc[25585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:06 dist-test-slave-djjw krb5kdc[25585](info): set up 2 sockets
Sep 28 11:03:06 dist-test-slave-djjw krb5kdc[25585](info): commencing operation
krb5kdc: starting...
W20260928 11:03:07.634377 4081 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:07 dist-test-slave-djjw krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:03:07.893452 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:08.231319 25598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:08.231649 25598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:08.231743 25598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:08.239194 25598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:08.239318 25598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:08.239367 25598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:08.239426 25598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:08.239470 25598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:08.250206 25598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:08.251603 25598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:08.252929 25598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:08.264448 25603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:08.264544 25606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:08.264568 25604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:08.265659 25598 server_base.cc:1034] running on GCE node
I20260928 11:03:08.266520 25598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:08.268018 25598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:08.269233 25598 hybrid_clock.cc:648] HybridClock initialized: now 1790593388269176 us; error 49 us; skew 500 ppm
I20260928 11:03:08.269582 25598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:08 dist-test-slave-djjw krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:08.272651 25598 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:03:08.274194 25598 webserver.cc:460] Webserver started at http://127.0.0.1:40067/ using document root <none> and password file <none>
I20260928 11:03:08.274883 25598 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:08.275012 25598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:08.275300 25598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:08.278450 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "e7e9371d126d4c28b9b1a55bf736a665"
format_stamp: "Formatted at 2026-09-28 11:03:08 on dist-test-slave-djjw"
server_key: "47011560ef20a044220d7e1fce34806c"
server_key_iv: "7529f1ec1049b8dc15c599b0bbd6c5b2"
server_key_version: "encryptionkey@0"
I20260928 11:03:08.279230 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "e7e9371d126d4c28b9b1a55bf736a665"
format_stamp: "Formatted at 2026-09-28 11:03:08 on dist-test-slave-djjw"
server_key: "47011560ef20a044220d7e1fce34806c"
server_key_iv: "7529f1ec1049b8dc15c599b0bbd6c5b2"
server_key_version: "encryptionkey@0"
I20260928 11:03:08.284837 25598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 11:03:08.288661 25613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:08.290164 25598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:03:08.290360 25598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "e7e9371d126d4c28b9b1a55bf736a665"
format_stamp: "Formatted at 2026-09-28 11:03:08 on dist-test-slave-djjw"
server_key: "47011560ef20a044220d7e1fce34806c"
server_key_iv: "7529f1ec1049b8dc15c599b0bbd6c5b2"
server_key_version: "encryptionkey@0"
I20260928 11:03:08.290560 25598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:08.311492 25598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:08.316012 25598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:08.316299 25598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:08.333940 25598 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39987
I20260928 11:03:08.333959 25664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39987 every 8 connection(s)
I20260928 11:03:08.335283 25598 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:08.338617 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25598
I20260928 11:03:08.338779 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:08.339066 25665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:08.339223 4081 external_mini_cluster.cc:1428] Setting key 6d2b3f4ac50a8a6e08275435e41eaa46
I20260928 11:03:08.354174 25665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665: Bootstrap starting.
I20260928 11:03:08.354228 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:08.340887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:33169 (local address 127.0.0.1:39987)
0928 11:03:08.341740 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:08.341755 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:08.341817 (+ 62us) server_negotiation.cc:407] Connection header received
0928 11:03:08.343736 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:08.343765 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:08.344297 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:08.344722 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:08.345258 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:08.347277 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:08.348756 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:08.349815 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:08.350045 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:08.351297 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:08.351311 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:08.351333 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:08.352104 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:08.352225 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:08.352260 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:08.352415 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:08.352701 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 11:03:08.352881 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":329,"threads_started":1}
I20260928 11:03:08.357875 25665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:08.358523 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:08.359073 25665 log.cc:824] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:08.362329 25665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665: No bootstrap required, opened a new log
I20260928 11:03:08.369958 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:08.363322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:34317 (local address 127.0.0.1:39987)
0928 11:03:08.363509 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:08.363515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:08.363848 (+ 333us) server_negotiation.cc:407] Connection header received
0928 11:03:08.363996 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:08.364002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:08.364084 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:08.364228 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:08.364689 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:08.365878 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:08.367175 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:08.367935 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:08.368062 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:08.369158 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:08.369164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:08.369169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:08.369199 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:08.369261 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:08.369285 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:08.369400 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:08.369521 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 11:03:08.369576 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 11:03:08.372169 25665 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e9371d126d4c28b9b1a55bf736a665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260928 11:03:08.372526 25665 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:08.372601 25665 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9371d126d4c28b9b1a55bf736a665, State: Initialized, Role: FOLLOWER
I20260928 11:03:08.373095 25665 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e9371d126d4c28b9b1a55bf736a665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260928 11:03:08.373302 25665 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:08.373395 25665 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:08.373519 25665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:08.375653 25665 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e9371d126d4c28b9b1a55bf736a665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260928 11:03:08.376132 25665 leader_election.cc:302] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e9371d126d4c28b9b1a55bf736a665; no voters:
I20260928 11:03:08.376718 25665 leader_election.cc:288] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:08.376852 25675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:08.377851 25675 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [term 1 LEADER]: Becoming Leader. State: Replica: e7e9371d126d4c28b9b1a55bf736a665, State: Running, Role: LEADER
I20260928 11:03:08.378391 25675 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e9371d126d4c28b9b1a55bf736a665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260928 11:03:08.378880 25665 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:08.381110 25676 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7e9371d126d4c28b9b1a55bf736a665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e9371d126d4c28b9b1a55bf736a665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } } }
I20260928 11:03:08.381340 25677 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e9371d126d4c28b9b1a55bf736a665. Latest consensus state: current_term: 1 leader_uuid: "e7e9371d126d4c28b9b1a55bf736a665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e9371d126d4c28b9b1a55bf736a665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } } }
I20260928 11:03:08.381625 25676 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:08.381631 25677 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665 [sys.catalog]: This master's current role is: LEADER
W20260928 11:03:08.388612 25691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 11:03:08.388744 25691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 11:03:08.388895 25684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:08.391201 25684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:08.400631 25684 catalog_manager.cc:1347] Generated new cluster ID: 6a882fc46cbd48bfb2bd5d43647ff2f4
I20260928 11:03:08.400768 25684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:08.416656 25684 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:08.418594 25684 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:08.435022 25684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7e9371d126d4c28b9b1a55bf736a665: Generated new TSK 0
I20260928 11:03:08.435981 25684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:08.443415 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25598
W20260928 11:03:08.444573 25660 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:33169
W20260928 11:03:08.445420 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25598: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:08.445560 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25598
W20260928 11:03:08.459437 25583 connection.cc:535] client connection to 127.0.0.1:39987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39987): Connection reset by peer (error 104)
2026-09-28T11:03:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:08 dist-test-slave-djjw krb5kdc[25696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:08 dist-test-slave-djjw krb5kdc[25696](info): set up 2 sockets
Sep 28 11:03:08 dist-test-slave-djjw krb5kdc[25696](info): commencing operation
krb5kdc: starting...
W20260928 11:03:09.462350 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:09 dist-test-slave-djjw krb5kdc[25696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 11:03:09.720481 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:10.046358 25709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:10.046676 25709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:10.046813 25709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:10.054610 25709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:10.054744 25709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:10.054796 25709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:10.054876 25709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:10.054929 25709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:10.065214 25709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:10.066634 25709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:10.067957 25709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:10.079617 25714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:10.079717 25715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:10.079808 25717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:10.080689 25709 server_base.cc:1034] running on GCE node
I20260928 11:03:10.081409 25709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:10.082876 25709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:10.084086 25709 hybrid_clock.cc:648] HybridClock initialized: now 1790593390084040 us; error 39 us; skew 500 ppm
I20260928 11:03:10.084420 25709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:10 dist-test-slave-djjw krb5kdc[25696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:10.092006 25709 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 11:03:10.093633 25709 webserver.cc:460] Webserver started at http://127.0.0.1:42397/ using document root <none> and password file <none>
I20260928 11:03:10.094352 25709 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:10.094467 25709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:10.094769 25709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:10.097939 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "f3c3c69d8cbc47549d2e1210a41758af"
format_stamp: "Formatted at 2026-09-28 11:03:10 on dist-test-slave-djjw"
server_key: "0f1cdd7e140d76d4e956e657fe7bc7db"
server_key_iv: "c40ba1ca46accd39951fa03255b5c4cd"
server_key_version: "encryptionkey@0"
I20260928 11:03:10.098735 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "f3c3c69d8cbc47549d2e1210a41758af"
format_stamp: "Formatted at 2026-09-28 11:03:10 on dist-test-slave-djjw"
server_key: "0f1cdd7e140d76d4e956e657fe7bc7db"
server_key_iv: "c40ba1ca46accd39951fa03255b5c4cd"
server_key_version: "encryptionkey@0"
I20260928 11:03:10.104115 25709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 11:03:10.107710 25724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:10.109059 25709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 11:03:10.109215 25709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "f3c3c69d8cbc47549d2e1210a41758af"
format_stamp: "Formatted at 2026-09-28 11:03:10 on dist-test-slave-djjw"
server_key: "0f1cdd7e140d76d4e956e657fe7bc7db"
server_key_iv: "c40ba1ca46accd39951fa03255b5c4cd"
server_key_version: "encryptionkey@0"
I20260928 11:03:10.109413 25709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:10.115125 25709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:10.119316 25709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:10.119626 25709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:10.137956 25709 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40255
I20260928 11:03:10.138002 25775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40255 every 8 connection(s)
I20260928 11:03:10.139415 25709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:10.143164 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:10.144667 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25709
I20260928 11:03:10.144855 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:10.145392 4081 external_mini_cluster.cc:1428] Setting key 2536f7543e275cfec37ccc7dd451edf1
I20260928 11:03:10.156632 25776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af: Bootstrap starting.
I20260928 11:03:10.158818 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:10.152768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:53731 (local address 127.0.0.1:40255)
0928 11:03:10.153531 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:10.153543 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:10.153585 (+ 42us) server_negotiation.cc:407] Connection header received
0928 11:03:10.155365 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:10.155389 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:10.155740 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:10.156065 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:10.157196 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:10.157211 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:10.157226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:10.157252 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:10.157318 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:10.157340 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:10.157519 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:10.157629 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 11:03:10.157871 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":344,"threads_started":1}
I20260928 11:03:10.160341 25776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:10.161520 25776 log.cc:824] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:10.162777 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:10.164402 25776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af: No bootstrap required, opened a new log
I20260928 11:03:10.169663 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:10.167324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:49367 (local address 127.0.0.1:40255)
0928 11:03:10.167516 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:10.167521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:10.167797 (+ 276us) server_negotiation.cc:407] Connection header received
0928 11:03:10.167969 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:10.167973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:10.168026 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:10.168105 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:10.169009 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:10.169013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:10.169017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:10.169035 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:10.169077 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:10.169091 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:10.169187 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:10.169294 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 11:03:10.169353 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 11:03:10.173238 25776 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3c69d8cbc47549d2e1210a41758af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260928 11:03:10.173658 25776 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:10.173734 25776 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c3c69d8cbc47549d2e1210a41758af, State: Initialized, Role: FOLLOWER
I20260928 11:03:10.174311 25776 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3c69d8cbc47549d2e1210a41758af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260928 11:03:10.174536 25776 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:10.174664 25776 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:10.174821 25776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:10.177191 25776 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3c69d8cbc47549d2e1210a41758af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260928 11:03:10.177662 25776 leader_election.cc:302] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c3c69d8cbc47549d2e1210a41758af; no voters:
I20260928 11:03:10.178300 25776 leader_election.cc:288] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:10.178471 25786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:10.179061 25786 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [term 1 LEADER]: Becoming Leader. State: Replica: f3c3c69d8cbc47549d2e1210a41758af, State: Running, Role: LEADER
I20260928 11:03:10.179631 25786 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3c69d8cbc47549d2e1210a41758af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260928 11:03:10.180317 25776 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:10.182421 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3c3c69d8cbc47549d2e1210a41758af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3c69d8cbc47549d2e1210a41758af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } } }
I20260928 11:03:10.182834 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:10.183274 25788 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c3c69d8cbc47549d2e1210a41758af. Latest consensus state: current_term: 1 leader_uuid: "f3c3c69d8cbc47549d2e1210a41758af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3c69d8cbc47549d2e1210a41758af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } } }
I20260928 11:03:10.183562 25788 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:10.183677 25792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:10.189750 25792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:10.199079 25792 catalog_manager.cc:1347] Generated new cluster ID: ebcbf07567fa4e42ae281b09733527a9
I20260928 11:03:10.199203 25792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:10.215926 25792 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:10.216358 25792 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:10.223021 25792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3c3c69d8cbc47549d2e1210a41758af: Generated new TSK 0
I20260928 11:03:10.223861 25792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:10.235000 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25709
W20260928 11:03:10.236065 25771 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:53731
W20260928 11:03:10.236763 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25709: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:10.236898 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25709
2026-09-28T11:03:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1791 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:10 dist-test-slave-djjw krb5kdc[25807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:10 dist-test-slave-djjw krb5kdc[25807](info): set up 2 sockets
Sep 28 11:03:10 dist-test-slave-djjw krb5kdc[25807](info): commencing operation
krb5kdc: starting...
W20260928 11:03:11.254462 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:11 dist-test-slave-djjw krb5kdc[25807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:11.504432 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41803
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:41803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:11.830574 25820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:11.830883 25820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:11.830979 25820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:11.838933 25820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:11.839064 25820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:11.839110 25820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:11.839174 25820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:11.839217 25820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:11.849705 25820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:41803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:41803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:11.851023 25820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:11.852300 25820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:11.863703 25825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:11.863685 25826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:11.864017 25828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:11.864503 25820 server_base.cc:1034] running on GCE node
I20260928 11:03:11.865367 25820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:11.866873 25820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:11.868113 25820 hybrid_clock.cc:648] HybridClock initialized: now 1790593391868053 us; error 51 us; skew 500 ppm
I20260928 11:03:11.868482 25820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:11 dist-test-slave-djjw krb5kdc[25807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:11.871670 25820 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:11.873107 25820 webserver.cc:460] Webserver started at http://127.3.252.126:44693/ using document root <none> and password file <none>
I20260928 11:03:11.873778 25820 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:11.873932 25820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:11.874229 25820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:11.877321 25820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "01c3c894403d48b4bcd4f9183aa2100d"
format_stamp: "Formatted at 2026-09-28 11:03:11 on dist-test-slave-djjw"
server_key: "ba3389e27918759e6c57b9c7e72239a6"
server_key_iv: "a2364e4d15cc29dfd7884e6c383a8aec"
server_key_version: "encryptionkey@0"
I20260928 11:03:11.878139 25820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "01c3c894403d48b4bcd4f9183aa2100d"
format_stamp: "Formatted at 2026-09-28 11:03:11 on dist-test-slave-djjw"
server_key: "ba3389e27918759e6c57b9c7e72239a6"
server_key_iv: "a2364e4d15cc29dfd7884e6c383a8aec"
server_key_version: "encryptionkey@0"
I20260928 11:03:11.883275 25820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 11:03:11.886812 25835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:11.888232 25820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 11:03:11.888403 25820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "01c3c894403d48b4bcd4f9183aa2100d"
format_stamp: "Formatted at 2026-09-28 11:03:11 on dist-test-slave-djjw"
server_key: "ba3389e27918759e6c57b9c7e72239a6"
server_key_iv: "a2364e4d15cc29dfd7884e6c383a8aec"
server_key_version: "encryptionkey@0"
I20260928 11:03:11.888592 25820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:11.926736 25820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:11.931234 25820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:11.931517 25820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:11.949433 25820 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:41803
I20260928 11:03:11.949455 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:41803 every 8 connection(s)
I20260928 11:03:11.950923 25820 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:11.954649 25887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:11.958910 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25820
I20260928 11:03:11.959088 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:11.959530 4081 external_mini_cluster.cc:1428] Setting key 9019a3c853325fb4467d93edcd08138c
I20260928 11:03:11.965858 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d: Bootstrap starting.
I20260928 11:03:11.969578 25887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:11.970820 25887 log.cc:824] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:11.974200 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d: No bootstrap required, opened a new log
Sep 28 11:03:11 dist-test-slave-djjw krb5kdc[25807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:03:11.984560 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:11.961338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48061 (local address 127.3.252.126:41803)
0928 11:03:11.962168 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:11.962180 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:11.962227 (+ 47us) server_negotiation.cc:407] Connection header received
0928 11:03:11.964305 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:11.964333 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:11.964764 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:11.965109 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:11.966847 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:11.968282 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:11.970019 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:11.970728 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:11.970863 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:11.975496 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:11.975507 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:11.975527 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:11.975573 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:11.979654 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:11.980537 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:11.980546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:11.980555 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:11.980764 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:11.981170 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:11.981175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:11.981178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:11.981946 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:11.982284 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:11.982696 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 11:03:11.983115 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":385,"threads_started":1}
I20260928 11:03:11.984532 25887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3c894403d48b4bcd4f9183aa2100d" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41803 } }
I20260928 11:03:11.985051 25887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:11.985153 25887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c3c894403d48b4bcd4f9183aa2100d, State: Initialized, Role: FOLLOWER
I20260928 11:03:11.985801 25887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3c894403d48b4bcd4f9183aa2100d" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41803 } }
I20260928 11:03:11.986128 25887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:11.986313 25887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:11.986505 25887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:11.989099 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:11.989471 25887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3c894403d48b4bcd4f9183aa2100d" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41803 } }
I20260928 11:03:11.990103 25887 leader_election.cc:302] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c3c894403d48b4bcd4f9183aa2100d; no voters:
I20260928 11:03:11.990793 25887 leader_election.cc:288] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:11.990911 25892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:11.992038 25892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [term 1 LEADER]: Becoming Leader. State: Replica: 01c3c894403d48b4bcd4f9183aa2100d, State: Running, Role: LEADER
I20260928 11:03:11.992810 25892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3c894403d48b4bcd4f9183aa2100d" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41803 } }
I20260928 11:03:11.993541 25887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:11.996104 25894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c3c894403d48b4bcd4f9183aa2100d. Latest consensus state: current_term: 1 leader_uuid: "01c3c894403d48b4bcd4f9183aa2100d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3c894403d48b4bcd4f9183aa2100d" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41803 } } }
I20260928 11:03:11.996527 25894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:11.997074 25893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c3c894403d48b4bcd4f9183aa2100d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3c894403d48b4bcd4f9183aa2100d" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 41803 } } }
I20260928 11:03:11.997365 25893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:12.002873 25901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:12.015887 25901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:12.027480 25901 catalog_manager.cc:1347] Generated new cluster ID: c61b5af56c504b7987cf061b1318065c
I20260928 11:03:12.027674 25901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:12.029299 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:12.004917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56689 (local address 127.3.252.126:41803)
0928 11:03:12.005114 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:12.005120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:12.011554 (+ 6434us) server_negotiation.cc:407] Connection header received
0928 11:03:12.012067 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:12.012074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:12.012162 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:12.012268 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:12.014379 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:12.015604 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:12.017512 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:12.018216 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:12.018317 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:12.021815 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:12.021820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:12.021823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:12.021894 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:12.025037 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:12.026086 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:12.026091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:12.026095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:12.026247 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:12.026830 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:12.026835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:12.026838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:12.027286 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:12.028537 (+ 1251us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:12.028719 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 11:03:12.028919 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 11:03:12.056818 25901 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:12.058813 25901 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:12.071839 25901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c3c894403d48b4bcd4f9183aa2100d: Generated new TSK 0
I20260928 11:03:12.072906 25901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:12.086355 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25820
I20260928 11:03:12.087729 25882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:12.186661 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25820
W20260928 11:03:12.202451 25805 connection.cc:535] client connection to 127.3.252.126:41803 recv error: Network error: failed to read from TLS socket (remote: 127.3.252.126:41803): Connection reset by peer (error 104)
2026-09-28T11:03:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:12 dist-test-slave-djjw krb5kdc[25919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:12 dist-test-slave-djjw krb5kdc[25919](info): set up 2 sockets
Sep 28 11:03:12 dist-test-slave-djjw krb5kdc[25919](info): commencing operation
krb5kdc: starting...
W20260928 11:03:13.222434 4081 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:13 dist-test-slave-djjw krb5kdc[25919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:13.468832 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40613
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:13.791491 25932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:13.791779 25932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:13.791903 25932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:13.799194 25932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:13.799314 25932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:13.799391 25932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:13.799469 25932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:13.799530 25932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:13.809717 25932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:40613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:13.811129 25932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:13.812448 25932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:13.823875 25937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:13.823906 25940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:13.823879 25938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:13.824692 25932 server_base.cc:1034] running on GCE node
I20260928 11:03:13.825438 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:13.826972 25932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:13.828207 25932 hybrid_clock.cc:648] HybridClock initialized: now 1790593393828155 us; error 44 us; skew 500 ppm
I20260928 11:03:13.828563 25932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:13 dist-test-slave-djjw krb5kdc[25919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:13.831645 25932 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:13.833139 25932 webserver.cc:460] Webserver started at http://127.3.252.126:46181/ using document root <none> and password file <none>
I20260928 11:03:13.833814 25932 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:13.833990 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:13.834283 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:13.837347 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "82448a76afd2403dacc4170df30e8cef"
format_stamp: "Formatted at 2026-09-28 11:03:13 on dist-test-slave-djjw"
server_key: "9a7b6ad0feddaf582650faa364eb1a82"
server_key_iv: "9332b97157cd9406c3b5f20a6d464832"
server_key_version: "encryptionkey@0"
I20260928 11:03:13.838172 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "82448a76afd2403dacc4170df30e8cef"
format_stamp: "Formatted at 2026-09-28 11:03:13 on dist-test-slave-djjw"
server_key: "9a7b6ad0feddaf582650faa364eb1a82"
server_key_iv: "9332b97157cd9406c3b5f20a6d464832"
server_key_version: "encryptionkey@0"
I20260928 11:03:13.843312 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 11:03:13.846814 25947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:13.848193 25932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 11:03:13.848352 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "82448a76afd2403dacc4170df30e8cef"
format_stamp: "Formatted at 2026-09-28 11:03:13 on dist-test-slave-djjw"
server_key: "9a7b6ad0feddaf582650faa364eb1a82"
server_key_iv: "9332b97157cd9406c3b5f20a6d464832"
server_key_version: "encryptionkey@0"
I20260928 11:03:13.848533 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:13.862567 25932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:13.866571 25932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:13.866825 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:13.884064 25932 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:40613
I20260928 11:03:13.884068 25998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:40613 every 8 connection(s)
I20260928 11:03:13.885430 25932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:13.889115 25999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:13.892701 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 25932
I20260928 11:03:13.892899 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:13.893424 4081 external_mini_cluster.cc:1428] Setting key b05140fad4f785720c7ad0894ec130a8
I20260928 11:03:13.901222 25999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef: Bootstrap starting.
I20260928 11:03:13.904441 25999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:13.905610 25999 log.cc:824] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:13.906261 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:13.895349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33781 (local address 127.3.252.126:40613)
0928 11:03:13.896148 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:13.896161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:13.896201 (+ 40us) server_negotiation.cc:407] Connection header received
0928 11:03:13.897817 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:13.897869 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:13.898299 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:13.898660 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:13.899220 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:13.900493 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:13.901803 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:13.902518 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:13.902711 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:13.903900 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:13.903914 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:13.903929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:13.904445 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:13.904516 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:13.904540 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:13.904680 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:13.904980 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 11:03:13.905158 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":363,"threads_started":1}
I20260928 11:03:13.908563 25999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef: No bootstrap required, opened a new log
I20260928 11:03:13.909924 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:13.916463 25999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82448a76afd2403dacc4170df30e8cef" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40613 } }
I20260928 11:03:13.916941 25999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:13.917042 25999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82448a76afd2403dacc4170df30e8cef, State: Initialized, Role: FOLLOWER
I20260928 11:03:13.917699 25999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82448a76afd2403dacc4170df30e8cef" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40613 } }
I20260928 11:03:13.918004 25999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:13.918155 25999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:13.918323 25999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:13.920539 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:13.914073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37953 (local address 127.3.252.126:40613)
0928 11:03:13.914270 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:13.914276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:13.914588 (+ 312us) server_negotiation.cc:407] Connection header received
0928 11:03:13.914744 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:13.914748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:13.914846 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:13.914930 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:13.915494 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:13.916279 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:13.918035 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:13.918563 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:13.918657 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:13.919785 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:13.919790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:13.919793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:13.919814 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:13.919876 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:13.919892 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:13.919970 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:13.920172 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 11:03:13.920225 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 11:03:13.921181 25999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82448a76afd2403dacc4170df30e8cef" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40613 } }
I20260928 11:03:13.921689 25999 leader_election.cc:302] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82448a76afd2403dacc4170df30e8cef; no voters:
I20260928 11:03:13.922226 25999 leader_election.cc:288] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:13.922341 26009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:13.922960 26009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [term 1 LEADER]: Becoming Leader. State: Replica: 82448a76afd2403dacc4170df30e8cef, State: Running, Role: LEADER
I20260928 11:03:13.923503 26009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82448a76afd2403dacc4170df30e8cef" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40613 } }
I20260928 11:03:13.924176 25999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:13.926136 26011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82448a76afd2403dacc4170df30e8cef. Latest consensus state: current_term: 1 leader_uuid: "82448a76afd2403dacc4170df30e8cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82448a76afd2403dacc4170df30e8cef" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40613 } } }
I20260928 11:03:13.926520 26011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:13.927389 26010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82448a76afd2403dacc4170df30e8cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82448a76afd2403dacc4170df30e8cef" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 40613 } } }
I20260928 11:03:13.927790 26010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:13.929126 26016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:13.933558 26016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:13.942754 26016 catalog_manager.cc:1347] Generated new cluster ID: 037a82b454884338acceb05d56ed2e17
I20260928 11:03:13.942893 26016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:13.966863 26016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:13.968741 26016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:13.982633 26016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82448a76afd2403dacc4170df30e8cef: Generated new TSK 0
I20260928 11:03:13.983939 26016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:14.000454 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25932
W20260928 11:03:14.001657 25994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33781
W20260928 11:03:14.002518 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 25932: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:14.002669 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 25932
W20260928 11:03:14.016579 25917 connection.cc:535] client connection to 127.3.252.126:40613 recv error: Network error: failed to read from TLS socket (remote: 127.3.252.126:40613): Connection reset by peer (error 104)
2026-09-28T11:03:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1814 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:14 dist-test-slave-djjw krb5kdc[26030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:14 dist-test-slave-djjw krb5kdc[26030](info): set up 2 sockets
Sep 28 11:03:14 dist-test-slave-djjw krb5kdc[26030](info): commencing operation
krb5kdc: starting...
W20260928 11:03:15.018411 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:15 dist-test-slave-djjw krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:15.267282 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:46247
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:46247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:15.600489 26043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:15.600788 26043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:15.600934 26043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:15.608340 26043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:15.608467 26043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:15.608523 26043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:15.608608 26043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:15.608664 26043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:15.618798 26043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:46247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:46247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:15.620133 26043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:15.621395 26043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:15.632978 26048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:15.632901 26049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:15.632936 26051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:15.634152 26043 server_base.cc:1034] running on GCE node
I20260928 11:03:15.635013 26043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:15.636493 26043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:15.637712 26043 hybrid_clock.cc:648] HybridClock initialized: now 1790593395637662 us; error 44 us; skew 500 ppm
I20260928 11:03:15.638123 26043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:15 dist-test-slave-djjw krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:15.645700 26043 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:15.647373 26043 webserver.cc:460] Webserver started at http://127.3.252.126:38501/ using document root <none> and password file <none>
I20260928 11:03:15.648077 26043 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:15.648201 26043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:15.648504 26043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:15.651671 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb"
format_stamp: "Formatted at 2026-09-28 11:03:15 on dist-test-slave-djjw"
server_key: "df21cd05e66f7229b759049e07a049bc"
server_key_iv: "d8299eb9d50b5d5de5f4e89002a6e040"
server_key_version: "encryptionkey@0"
I20260928 11:03:15.652441 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb"
format_stamp: "Formatted at 2026-09-28 11:03:15 on dist-test-slave-djjw"
server_key: "df21cd05e66f7229b759049e07a049bc"
server_key_iv: "d8299eb9d50b5d5de5f4e89002a6e040"
server_key_version: "encryptionkey@0"
I20260928 11:03:15.657779 26043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 11:03:15.661582 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:15.663159 26043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:03:15.663350 26043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb"
format_stamp: "Formatted at 2026-09-28 11:03:15 on dist-test-slave-djjw"
server_key: "df21cd05e66f7229b759049e07a049bc"
server_key_iv: "d8299eb9d50b5d5de5f4e89002a6e040"
server_key_version: "encryptionkey@0"
I20260928 11:03:15.663573 26043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:15.669582 26043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:15.673799 26043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:15.674155 26043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:15.692773 26043 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:46247
I20260928 11:03:15.692798 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:46247 every 8 connection(s)
I20260928 11:03:15.694257 26043 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:15.697932 26110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:15.702318 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26043
I20260928 11:03:15.702481 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:15.702884 4081 external_mini_cluster.cc:1428] Setting key f50be72fcc4558039d732eb42d8a6396
I20260928 11:03:15.709993 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb: Bootstrap starting.
I20260928 11:03:15.711790 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:15.704617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45215 (local address 127.3.252.126:46247)
0928 11:03:15.705432 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:15.705448 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:15.705501 (+ 53us) server_negotiation.cc:407] Connection header received
0928 11:03:15.708009 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:15.708044 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:15.708506 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:15.708952 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:15.709928 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:15.709936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:15.709954 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:15.709971 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:15.710051 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:15.710078 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:15.710278 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:15.710407 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 11:03:15.710598 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":322,"threads_started":1}
I20260928 11:03:15.713696 26110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:15.714901 26110 log.cc:824] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:15.715749 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:15.717800 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb: No bootstrap required, opened a new log
I20260928 11:03:15.722280 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:15.719977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52181 (local address 127.3.252.126:46247)
0928 11:03:15.720133 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:15.720137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:15.720432 (+ 295us) server_negotiation.cc:407] Connection header received
0928 11:03:15.720603 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:15.720607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:15.720656 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:15.720732 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:15.721663 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:15.721671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:15.721674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:15.721690 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:15.721728 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:15.721745 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:15.721898 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:15.721975 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 11:03:15.722020 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 11:03:15.726097 26110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46247 } }
I20260928 11:03:15.726495 26110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:15.726594 26110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fe7c5f6ca944d9ae5bf747fcd23aeb, State: Initialized, Role: FOLLOWER
I20260928 11:03:15.727198 26110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46247 } }
I20260928 11:03:15.727423 26110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:15.727551 26110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:15.727705 26110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:15.729913 26110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46247 } }
I20260928 11:03:15.730405 26110 leader_election.cc:302] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74fe7c5f6ca944d9ae5bf747fcd23aeb; no voters:
I20260928 11:03:15.731001 26110 leader_election.cc:288] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:15.731119 26120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:15.731719 26120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [term 1 LEADER]: Becoming Leader. State: Replica: 74fe7c5f6ca944d9ae5bf747fcd23aeb, State: Running, Role: LEADER
I20260928 11:03:15.732203 26120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46247 } }
I20260928 11:03:15.732960 26110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:15.735179 26122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74fe7c5f6ca944d9ae5bf747fcd23aeb. Latest consensus state: current_term: 1 leader_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46247 } } }
I20260928 11:03:15.735580 26122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:15.736824 26127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:15.736726 26121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fe7c5f6ca944d9ae5bf747fcd23aeb" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 46247 } } }
I20260928 11:03:15.737042 26121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:15.742174 26127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:15.751214 26127 catalog_manager.cc:1347] Generated new cluster ID: c2c917f6c1d9487ab5ae182eb11cc112
I20260928 11:03:15.751351 26127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:15.769274 26127 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:15.769714 26127 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:15.777244 26127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74fe7c5f6ca944d9ae5bf747fcd23aeb: Generated new TSK 0
I20260928 11:03:15.778167 26127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:15.786388 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26043
W20260928 11:03:15.787410 26105 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45215
W20260928 11:03:15.788075 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26043: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:15.788192 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26043
2026-09-28T11:03:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1785 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:15 dist-test-slave-djjw krb5kdc[26141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:15 dist-test-slave-djjw krb5kdc[26141](info): set up 2 sockets
Sep 28 11:03:15 dist-test-slave-djjw krb5kdc[26141](info): commencing operation
krb5kdc: starting...
W20260928 11:03:16.818472 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:16 dist-test-slave-djjw krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:17.070983 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36879
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:17.406735 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:17.407019 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:17.407152 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:17.414361 26154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:17.414480 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:17.414553 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:17.414635 26154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:17.414695 26154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:17.424624 26154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:17.425976 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:17.427322 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:17.438900 26159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:17.438879 26160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:17.438927 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:17.439906 26154 server_base.cc:1034] running on GCE node
I20260928 11:03:17.440708 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:17.442190 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:17.443378 26154 hybrid_clock.cc:648] HybridClock initialized: now 1790593397443328 us; error 47 us; skew 500 ppm
I20260928 11:03:17.443701 26154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:17 dist-test-slave-djjw krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593397, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:17.446923 26154 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:17.448482 26154 webserver.cc:460] Webserver started at http://127.3.252.126:44321/ using document root <none> and password file <none>
I20260928 11:03:17.449152 26154 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:17.449263 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:17.449582 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:17.452683 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "cd58cbd6134f491d98f52704fd5fc1ec"
format_stamp: "Formatted at 2026-09-28 11:03:17 on dist-test-slave-djjw"
server_key: "f61e656bf0f3b451dc49000ddae0f3b9"
server_key_iv: "36c60a780732305002d7ae81d0e59790"
server_key_version: "encryptionkey@0"
I20260928 11:03:17.453456 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "cd58cbd6134f491d98f52704fd5fc1ec"
format_stamp: "Formatted at 2026-09-28 11:03:17 on dist-test-slave-djjw"
server_key: "f61e656bf0f3b451dc49000ddae0f3b9"
server_key_iv: "36c60a780732305002d7ae81d0e59790"
server_key_version: "encryptionkey@0"
I20260928 11:03:17.458837 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 11:03:17.462615 26169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:17.464004 26154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:03:17.464186 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "cd58cbd6134f491d98f52704fd5fc1ec"
format_stamp: "Formatted at 2026-09-28 11:03:17 on dist-test-slave-djjw"
server_key: "f61e656bf0f3b451dc49000ddae0f3b9"
server_key_iv: "36c60a780732305002d7ae81d0e59790"
server_key_version: "encryptionkey@0"
I20260928 11:03:17.464403 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:17.498330 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:17.502568 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:17.502833 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:17.520354 26154 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:36879
I20260928 11:03:17.520385 26220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:36879 every 8 connection(s)
I20260928 11:03:17.521749 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:17.525408 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:17.531580 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26154
I20260928 11:03:17.531759 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:17.532303 4081 external_mini_cluster.cc:1428] Setting key dc344f41dad99e7bf6632a27f0cad993
I20260928 11:03:17.536369 26221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec: Bootstrap starting.
I20260928 11:03:17.539911 26221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:17.540998 26221 log.cc:824] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:17.544088 26221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec: No bootstrap required, opened a new log
Sep 28 11:03:17 dist-test-slave-djjw krb5kdc[26141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:03:17.553027 26221 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36879 } }
I20260928 11:03:17.553496 26221 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:17.553607 26221 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd58cbd6134f491d98f52704fd5fc1ec, State: Initialized, Role: FOLLOWER
I20260928 11:03:17.554270 26221 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36879 } }
I20260928 11:03:17.554533 26221 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:17.554688 26221 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:17.554843 26221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:17.555390 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:17.534198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58755 (local address 127.3.252.126:36879)
0928 11:03:17.535003 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:17.535014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:17.535054 (+ 40us) server_negotiation.cc:407] Connection header received
0928 11:03:17.536719 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:17.536743 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:17.537143 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:17.537452 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:17.539168 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:17.540560 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:17.542154 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:17.542864 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:17.542998 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:17.547719 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:17.547727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:17.547747 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:17.547814 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:17.551004 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:17.551786 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:17.551793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:17.551799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:17.551980 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:17.552395 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:17.552399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:17.552401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:17.552973 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:17.553194 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:17.553767 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 11:03:17.554206 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":374,"threads_started":1}
I20260928 11:03:17.557220 26221 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36879 } }
I20260928 11:03:17.557740 26221 leader_election.cc:302] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd58cbd6134f491d98f52704fd5fc1ec; no voters:
I20260928 11:03:17.558374 26221 leader_election.cc:288] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:17.558518 26226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:17.559232 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:17.560233 26221 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:17.566721 26226 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [term 1 LEADER]: Becoming Leader. State: Replica: cd58cbd6134f491d98f52704fd5fc1ec, State: Running, Role: LEADER
I20260928 11:03:17.567520 26226 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36879 } }
I20260928 11:03:17.578794 26227 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36879 } } }
I20260928 11:03:17.579206 26227 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:17.579181 26228 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd58cbd6134f491d98f52704fd5fc1ec. Latest consensus state: current_term: 1 leader_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd58cbd6134f491d98f52704fd5fc1ec" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36879 } } }
I20260928 11:03:17.579474 26228 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:17.580382 26247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:17.585456 26247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:17.590669 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:17.569540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44991 (local address 127.3.252.126:36879)
0928 11:03:17.574050 (+ 4510us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:17.574058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:17.574760 (+ 702us) server_negotiation.cc:407] Connection header received
0928 11:03:17.574993 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:17.574999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:17.575103 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:17.575235 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:17.577202 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:17.578097 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:17.579881 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:17.580650 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:17.580786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:17.583733 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:17.583739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:17.583745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:17.583784 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:17.587585 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:17.588439 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:17.588446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:17.588449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:17.588607 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:17.588997 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:17.589002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:17.589005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:17.589431 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:17.589677 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:17.590059 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 11:03:17.590285 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4348}
I20260928 11:03:17.594455 26247 catalog_manager.cc:1347] Generated new cluster ID: 0bd85bed2dcb42fc90f561a494a92edc
I20260928 11:03:17.594573 26247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:17.607053 26247 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:17.609020 26247 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:17.621654 26247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd58cbd6134f491d98f52704fd5fc1ec: Generated new TSK 0
I20260928 11:03:17.622844 26247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:17.637936 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26154
I20260928 11:03:17.640084 26216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:17.733201 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26154
2026-09-28T11:03:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:17 dist-test-slave-djjw krb5kdc[26254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:17 dist-test-slave-djjw krb5kdc[26254](info): set up 2 sockets
Sep 28 11:03:17 dist-test-slave-djjw krb5kdc[26254](info): commencing operation
krb5kdc: starting...
W20260928 11:03:18.758333 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:18 dist-test-slave-djjw krb5kdc[26254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:19.011113 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36925
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:19.340623 26267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:19.340945 26267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:19.341074 26267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:19.348528 26267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:19.348659 26267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:19.348712 26267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:19.348796 26267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:19.348850 26267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:19.359025 26267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:19.360364 26267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:19.361680 26267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:19.373570 26272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:19.373636 26275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:19.373641 26273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:19.374465 26267 server_base.cc:1034] running on GCE node
I20260928 11:03:19.375227 26267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:19.376662 26267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:19.377883 26267 hybrid_clock.cc:648] HybridClock initialized: now 1790593399377828 us; error 44 us; skew 500 ppm
I20260928 11:03:19.378254 26267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:19 dist-test-slave-djjw krb5kdc[26254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:19.381315 26267 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:19.382717 26267 webserver.cc:460] Webserver started at http://127.3.252.126:40545/ using document root <none> and password file <none>
I20260928 11:03:19.383338 26267 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:19.383440 26267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:19.383685 26267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:19.386823 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "b2851901ed42437cbbe62d87efc587e3"
format_stamp: "Formatted at 2026-09-28 11:03:19 on dist-test-slave-djjw"
server_key: "e1fbc47be50355df87dc741c2cc9d5ac"
server_key_iv: "bb0ff63f434b029331c61d7142291302"
server_key_version: "encryptionkey@0"
I20260928 11:03:19.387568 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "b2851901ed42437cbbe62d87efc587e3"
format_stamp: "Formatted at 2026-09-28 11:03:19 on dist-test-slave-djjw"
server_key: "e1fbc47be50355df87dc741c2cc9d5ac"
server_key_iv: "bb0ff63f434b029331c61d7142291302"
server_key_version: "encryptionkey@0"
I20260928 11:03:19.393040 26267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 11:03:19.396898 26282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:19.398430 26267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 11:03:19.398599 26267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "b2851901ed42437cbbe62d87efc587e3"
format_stamp: "Formatted at 2026-09-28 11:03:19 on dist-test-slave-djjw"
server_key: "e1fbc47be50355df87dc741c2cc9d5ac"
server_key_iv: "bb0ff63f434b029331c61d7142291302"
server_key_version: "encryptionkey@0"
I20260928 11:03:19.398784 26267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:19.415566 26267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:19.419886 26267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:19.420178 26267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:19.438197 26267 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:36925
I20260928 11:03:19.438201 26333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:36925 every 8 connection(s)
I20260928 11:03:19.439666 26267 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:19.443455 26334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:19.445978 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26267
I20260928 11:03:19.446177 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:19.446692 4081 external_mini_cluster.cc:1428] Setting key cbd1ee51cf297ff5adf65e3606e3ff86
I20260928 11:03:19.455971 26334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3: Bootstrap starting.
I20260928 11:03:19.459254 26334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:19.460376 26334 log.cc:824] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:19.460471 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:19.448808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51171 (local address 127.3.252.126:36925)
0928 11:03:19.449718 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:19.449736 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:19.449782 (+ 46us) server_negotiation.cc:407] Connection header received
0928 11:03:19.451640 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:19.451682 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:19.452160 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:19.452567 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:19.453170 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:19.454613 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:19.455966 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:19.456703 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:19.456871 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:19.458080 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:19.458088 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:19.458104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:19.458643 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:19.458725 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:19.458755 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:19.458857 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:19.459099 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 11:03:19.459396 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":385,"threads_started":1}
I20260928 11:03:19.463388 26334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3: No bootstrap required, opened a new log
I20260928 11:03:19.464370 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:19.472205 26334 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2851901ed42437cbbe62d87efc587e3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36925 } }
I20260928 11:03:19.472631 26334 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:19.472757 26334 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2851901ed42437cbbe62d87efc587e3, State: Initialized, Role: FOLLOWER
I20260928 11:03:19.473480 26334 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2851901ed42437cbbe62d87efc587e3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36925 } }
I20260928 11:03:19.473784 26334 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:19.473977 26334 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:19.474146 26334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:19.475253 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:19.468628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38825 (local address 127.3.252.126:36925)
0928 11:03:19.468830 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:19.468836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:19.469204 (+ 368us) server_negotiation.cc:407] Connection header received
0928 11:03:19.469364 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:19.469370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:19.469455 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:19.469563 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:19.470135 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:19.471280 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:19.472693 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:19.473316 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:19.473438 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:19.474521 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:19.474527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:19.474532 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:19.474557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:19.474603 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:19.474624 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:19.474724 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:19.474806 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 11:03:19.474913 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 11:03:19.477170 26334 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2851901ed42437cbbe62d87efc587e3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36925 } }
I20260928 11:03:19.477705 26334 leader_election.cc:302] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2851901ed42437cbbe62d87efc587e3; no voters:
I20260928 11:03:19.478278 26334 leader_election.cc:288] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:19.478416 26344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:19.479120 26344 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [term 1 LEADER]: Becoming Leader. State: Replica: b2851901ed42437cbbe62d87efc587e3, State: Running, Role: LEADER
I20260928 11:03:19.479719 26344 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2851901ed42437cbbe62d87efc587e3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36925 } }
I20260928 11:03:19.480330 26334 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:19.482581 26345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2851901ed42437cbbe62d87efc587e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2851901ed42437cbbe62d87efc587e3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36925 } } }
I20260928 11:03:19.482944 26345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:19.483371 26346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2851901ed42437cbbe62d87efc587e3. Latest consensus state: current_term: 1 leader_uuid: "b2851901ed42437cbbe62d87efc587e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2851901ed42437cbbe62d87efc587e3" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36925 } } }
I20260928 11:03:19.486258 26346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:19.487735 26351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:19.493959 26351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:19.508510 26351 catalog_manager.cc:1347] Generated new cluster ID: 0652dbfb1fa04746bb85fba1ba119d5e
I20260928 11:03:19.508649 26351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:19.532606 26351 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:19.534556 26351 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:19.553566 26351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2851901ed42437cbbe62d87efc587e3: Generated new TSK 0
I20260928 11:03:19.554513 26351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:19.570433 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26267
W20260928 11:03:19.571558 26329 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51171
W20260928 11:03:19.572257 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26267: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:19.572387 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26267
2026-09-28T11:03:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:19 dist-test-slave-djjw krb5kdc[26365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:19 dist-test-slave-djjw krb5kdc[26365](info): set up 2 sockets
Sep 28 11:03:19 dist-test-slave-djjw krb5kdc[26365](info): commencing operation
krb5kdc: starting...
W20260928 11:03:20.618362 4081 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:20 dist-test-slave-djjw krb5kdc[26365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:20.869616 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38135
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:38135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:21.197103 26378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:21.197389 26378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:21.197486 26378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:21.204864 26378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:21.204982 26378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:21.205036 26378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:21.205096 26378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:21.205137 26378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:21.215368 26378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:38135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:38135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:21.216635 26378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:21.217890 26378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:21.229431 26383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:21.229467 26386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:21.229517 26384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:21.230098 26378 server_base.cc:1034] running on GCE node
I20260928 11:03:21.230926 26378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:21.232393 26378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:21.233632 26378 hybrid_clock.cc:648] HybridClock initialized: now 1790593401233592 us; error 28 us; skew 500 ppm
I20260928 11:03:21.234045 26378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:21 dist-test-slave-djjw krb5kdc[26365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593401, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:21.241851 26378 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:21.243424 26378 webserver.cc:460] Webserver started at http://127.3.252.126:36637/ using document root <none> and password file <none>
I20260928 11:03:21.244102 26378 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:21.244205 26378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:21.244467 26378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:21.247543 26378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "ebefd6d5b43b466e9b4c19634dc981fc"
format_stamp: "Formatted at 2026-09-28 11:03:21 on dist-test-slave-djjw"
server_key: "ddf58f2157e6faf4d4a3c130effcf966"
server_key_iv: "f7273b09c0fd7320fb57261d4d19e82e"
server_key_version: "encryptionkey@0"
I20260928 11:03:21.248302 26378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "ebefd6d5b43b466e9b4c19634dc981fc"
format_stamp: "Formatted at 2026-09-28 11:03:21 on dist-test-slave-djjw"
server_key: "ddf58f2157e6faf4d4a3c130effcf966"
server_key_iv: "f7273b09c0fd7320fb57261d4d19e82e"
server_key_version: "encryptionkey@0"
I20260928 11:03:21.253656 26378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 11:03:21.257427 26393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:21.259030 26378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:03:21.259210 26378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "ebefd6d5b43b466e9b4c19634dc981fc"
format_stamp: "Formatted at 2026-09-28 11:03:21 on dist-test-slave-djjw"
server_key: "ddf58f2157e6faf4d4a3c130effcf966"
server_key_iv: "f7273b09c0fd7320fb57261d4d19e82e"
server_key_version: "encryptionkey@0"
I20260928 11:03:21.259413 26378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:21.265251 26378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:21.269443 26378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:21.269757 26378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:21.287954 26378 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:38135
I20260928 11:03:21.288030 26444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:38135 every 8 connection(s)
I20260928 11:03:21.289446 26378 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:21.293237 26445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:21.293381 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26378
I20260928 11:03:21.293533 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:21.293998 4081 external_mini_cluster.cc:1428] Setting key f7dfa50b7dccd0defe89eb1ac5d6d34c
I20260928 11:03:21.302836 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:21.295836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35275 (local address 127.3.252.126:38135)
0928 11:03:21.296729 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:21.296742 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:21.296785 (+ 43us) server_negotiation.cc:407] Connection header received
0928 11:03:21.299247 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:21.299276 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:21.299698 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:21.300082 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:21.301231 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:21.301238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:21.301253 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:21.301268 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:21.301329 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:21.301353 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:21.301520 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:21.301641 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 11:03:21.301865 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":400,"threads_started":1}
I20260928 11:03:21.306553 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:21.306692 26445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc: Bootstrap starting.
I20260928 11:03:21.310822 26445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:21.312335 26445 log.cc:824] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:21.314567 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:21.311978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42347 (local address 127.3.252.126:38135)
0928 11:03:21.312183 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:21.312189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:21.312597 (+ 408us) server_negotiation.cc:407] Connection header received
0928 11:03:21.312768 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:21.312774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:21.312851 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:21.312990 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:21.313950 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:21.313955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:21.313959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 11:03:21.313978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:21.314024 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 11:03:21.314056 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:21.314185 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:21.314234 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 11:03:21.314295 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 11:03:21.315675 26445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc: No bootstrap required, opened a new log
I20260928 11:03:21.322999 26445 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38135 } }
I20260928 11:03:21.323396 26445 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 11:03:21.323495 26445 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebefd6d5b43b466e9b4c19634dc981fc, State: Initialized, Role: FOLLOWER
I20260928 11:03:21.324066 26445 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38135 } }
I20260928 11:03:21.324298 26445 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 11:03:21.324430 26445 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 11:03:21.324584 26445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:21.326849 26445 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38135 } }
I20260928 11:03:21.327420 26445 leader_election.cc:302] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebefd6d5b43b466e9b4c19634dc981fc; no voters:
I20260928 11:03:21.328037 26445 leader_election.cc:288] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 11:03:21.328176 26455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:21.328789 26455 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [term 1 LEADER]: Becoming Leader. State: Replica: ebefd6d5b43b466e9b4c19634dc981fc, State: Running, Role: LEADER
I20260928 11:03:21.329293 26455 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38135 } }
I20260928 11:03:21.330051 26445 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:21.332212 26457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebefd6d5b43b466e9b4c19634dc981fc. Latest consensus state: current_term: 1 leader_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38135 } } }
I20260928 11:03:21.332682 26457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:21.335093 26464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:21.335002 26456 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebefd6d5b43b466e9b4c19634dc981fc" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 38135 } } }
I20260928 11:03:21.335397 26456 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:21.339392 26464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:21.347887 26464 catalog_manager.cc:1347] Generated new cluster ID: 6d1fee4e729945c690f5eed293f6244d
I20260928 11:03:21.348007 26464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:21.363487 26464 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:21.364046 26464 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:21.377007 26464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebefd6d5b43b466e9b4c19634dc981fc: Generated new TSK 0
I20260928 11:03:21.378237 26464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:21.394101 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26378
W20260928 11:03:21.395210 26440 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35275
W20260928 11:03:21.395897 4081 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26378: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 11:03:21.396026 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26378
2026-09-28T11:03:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1821 ms)
[----------] 18 tests from AuthTokenIssuingTest (33216 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:21 dist-test-slave-djjw krb5kdc[26476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:21 dist-test-slave-djjw krb5kdc[26476](info): set up 2 sockets
Sep 28 11:03:21 dist-test-slave-djjw krb5kdc[26476](info): commencing operation
krb5kdc: starting...
W20260928 11:03:22.414448 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:22 dist-test-slave-djjw krb5kdc[26476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:22.671568 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:32783
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:32783,127.3.252.125:34717,127.3.252.124:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:22.998701 26489 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 11:03:22.999083 26489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:22.999181 26489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:22.999295 26489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:23.006716 26489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:23.006826 26489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:23.006906 26489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:23.006976 26489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:23.007059 26489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:23.017166 26489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:32783,127.3.252.125:34717,127.3.252.124:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:32783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:23.018537 26489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:23.019848 26489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:23.031365 26495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:23.031445 26494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:23.031415 26497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:23.032362 26489 server_base.cc:1034] running on GCE node
I20260928 11:03:23.033154 26489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:23.034677 26489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:23.035889 26489 hybrid_clock.cc:648] HybridClock initialized: now 1790593403035821 us; error 42 us; skew 500 ppm
I20260928 11:03:23.036253 26489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:23 dist-test-slave-djjw krb5kdc[26476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:23.039565 26489 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:23.041175 26489 webserver.cc:460] Webserver started at http://127.3.252.126:40495/ using document root <none> and password file <none>
I20260928 11:03:23.041951 26489 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:23.042054 26489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:23.042383 26489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:23.045485 26489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "069c9505661842ec87e0b2753d0c1c00"
format_stamp: "Formatted at 2026-09-28 11:03:23 on dist-test-slave-djjw"
server_key: "3af8a370ec80c3665d6f9f150bea01d7"
server_key_iv: "0baf58b3b63b6a7bef0713592727d5da"
server_key_version: "encryptionkey@0"
I20260928 11:03:23.046308 26489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "069c9505661842ec87e0b2753d0c1c00"
format_stamp: "Formatted at 2026-09-28 11:03:23 on dist-test-slave-djjw"
server_key: "3af8a370ec80c3665d6f9f150bea01d7"
server_key_iv: "0baf58b3b63b6a7bef0713592727d5da"
server_key_version: "encryptionkey@0"
I20260928 11:03:23.051571 26489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 11:03:23.055159 26504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:23.056559 26489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:03:23.056728 26489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "069c9505661842ec87e0b2753d0c1c00"
format_stamp: "Formatted at 2026-09-28 11:03:23 on dist-test-slave-djjw"
server_key: "3af8a370ec80c3665d6f9f150bea01d7"
server_key_iv: "0baf58b3b63b6a7bef0713592727d5da"
server_key_version: "encryptionkey@0"
I20260928 11:03:23.056931 26489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:23.083711 26489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:23.088176 26489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:23.088435 26489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:23.105752 26489 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:32783
I20260928 11:03:23.105788 26555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:32783 every 8 connection(s)
I20260928 11:03:23.107102 26489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:23.110944 26556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:23.115748 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26489
I20260928 11:03:23.115972 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:23.116827 4081 external_mini_cluster.cc:1428] Setting key 10d2895ac6aae94c7745b53f21c02bfd
I20260928 11:03:23.118983 26556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:23 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.125@KRBTEST.COM; defaulting to no policy
I20260928 11:03:23.141681 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.123119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:32783 (local address 127.0.0.1:41122)
0928 11:03:23.124044 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.124099 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:23.126218 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:23.127055 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:23.127062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:23.127786 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:23.128138 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.128157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.129667 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.129671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.130802 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.130815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.131731 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.131738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.131896 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.132504 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:23.132527 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:23.134820 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:23.137876 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.137888 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.137901 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.138298 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.138693 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.138698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.138702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.138851 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.139608 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:23.139615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:23.139821 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 11:03:23.140040 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 11:03:23.140783 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":188,"thread_start_us":427,"threads_started":1}
I20260928 11:03:23.141702 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.123098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.3.252.126:32783)
0928 11:03:23.124880 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:23.124888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:23.124945 (+ 57us) server_negotiation.cc:407] Connection header received
0928 11:03:23.126592 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:23.126598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:23.126839 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:23.127014 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:23.128334 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.129517 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.130935 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.131496 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.131677 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.135037 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:23.135047 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:23.135060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:23.135090 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:23.137731 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.138427 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.138434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.138436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.138581 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.138958 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.138962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.138964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.139475 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:23.139671 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:23.140197 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 11:03:23.140561 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":1573,"thread_start_us":1341,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.125@KRBTEST.COM" created.
I20260928 11:03:23.144532 26556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:03:23.145562 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.145166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34818)
0928 11:03:23.145338 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.145448 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 11:03:23.145779 26507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
I20260928 11:03:23.147280 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.146935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34824)
0928 11:03:23.147073 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.147146 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 11:03:23.147454 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:34717: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab.
Entry for principal kudu/127.3.252.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab.
I20260928 11:03:23.172411 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:34717
--webserver_interface=127.3.252.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab
--principal=kudu/127.3.252.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:32783,127.3.252.125:34717,127.3.252.124:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:03:23.196817 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } attempt: 1
I20260928 11:03:23.197990 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.197502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34832)
0928 11:03:23.197697 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.197821 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 11:03:23.199129 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.198730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34842)
0928 11:03:23.198896 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.198973 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 11:03:23.199306 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:34717: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
I20260928 11:03:23.267582 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } attempt: 2
I20260928 11:03:23.268932 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.268409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34858)
0928 11:03:23.268691 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.268786 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 11:03:23.270133 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.269702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34862)
0928 11:03:23.269903 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.269981 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 11:03:23.270339 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:34717: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
I20260928 11:03:23.361589 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } attempt: 3
I20260928 11:03:23.362880 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.362350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34866)
0928 11:03:23.362639 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.362734 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 11:03:23.363876 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.363515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34880)
0928 11:03:23.363664 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.363737 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 11:03:23.364046 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:34717: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
W20260928 11:03:23.505961 26563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 11:03:23.506333 26563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:23.506429 26563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:23.506544 26563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 11:03:23.507272 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } attempt: 4
I20260928 11:03:23.508493 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.507963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34890)
0928 11:03:23.508276 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.508366 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 11:03:23.509434 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.509082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34896)
0928 11:03:23.509229 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.509316 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 11:03:23.509619 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:34717: Network error: Client connection negotiation failed: client connection to 127.3.252.125:34717: connect: Connection refused (error 111)
W20260928 11:03:23.514375 26563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:23.514482 26563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:23.514572 26563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:23.514637 26563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:23.514719 26563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:23.524852 26563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:32783,127.3.252.125:34717,127.3.252.124:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab
--principal=kudu/127.3.252.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:34717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.252.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:23.526268 26563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:23.527597 26563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:23.539706 26571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:23.539742 26568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:23.539789 26569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:23.547778 26563 server_base.cc:1034] running on GCE node
I20260928 11:03:23.548681 26563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:23.550258 26563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:23.551489 26563 hybrid_clock.cc:648] HybridClock initialized: now 1790593403551427 us; error 43 us; skew 500 ppm
I20260928 11:03:23.551826 26563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:23 dist-test-slave-djjw krb5kdc[26476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:23.555051 26563 init.cc:377] Logged in from keytab as kudu/127.3.252.125@KRBTEST.COM (short username kudu)
I20260928 11:03:23.556624 26563 webserver.cc:460] Webserver started at http://127.3.252.125:37015/ using document root <none> and password file <none>
I20260928 11:03:23.557310 26563 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:23.557395 26563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:23.557673 26563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:23.560735 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data/instance:
uuid: "89fb661b65f7431d90249ca5a293610d"
format_stamp: "Formatted at 2026-09-28 11:03:23 on dist-test-slave-djjw"
server_key: "9eac9f5e4ef7dbf66d01b85d85234fe1"
server_key_iv: "6abeb2ae7b85abb4aff077d74d374e6f"
server_key_version: "encryptionkey@0"
I20260928 11:03:23.561487 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/wal/instance:
uuid: "89fb661b65f7431d90249ca5a293610d"
format_stamp: "Formatted at 2026-09-28 11:03:23 on dist-test-slave-djjw"
server_key: "9eac9f5e4ef7dbf66d01b85d85234fe1"
server_key_iv: "6abeb2ae7b85abb4aff077d74d374e6f"
server_key_version: "encryptionkey@0"
I20260928 11:03:23.566774 26563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 11:03:23.570508 26578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:23.571957 26563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 11:03:23.572113 26563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/wal
uuid: "89fb661b65f7431d90249ca5a293610d"
format_stamp: "Formatted at 2026-09-28 11:03:23 on dist-test-slave-djjw"
server_key: "9eac9f5e4ef7dbf66d01b85d85234fe1"
server_key_iv: "6abeb2ae7b85abb4aff077d74d374e6f"
server_key_version: "encryptionkey@0"
I20260928 11:03:23.572326 26563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:23.597343 26563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:23.601727 26563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:23.602025 26563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:23.619443 26563 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.125:34717
I20260928 11:03:23.619444 26629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.125:34717 every 8 connection(s)
I20260928 11:03:23.620826 26563 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
I20260928 11:03:23.624573 26630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:23.627619 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26563
I20260928 11:03:23.627813 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-1/wal/instance
I20260928 11:03:23.628399 4081 external_mini_cluster.cc:1428] Setting key b486b57464ddf1dc472b9277af0965cb
I20260928 11:03:23.633075 26630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:23 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.124@KRBTEST.COM; defaulting to no policy
I20260928 11:03:23.656243 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.636455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.3.252.126:32783)
0928 11:03:23.636678 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:23.636685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:23.637920 (+ 1235us) server_negotiation.cc:407] Connection header received
0928 11:03:23.640024 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:23.640030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:23.640109 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:23.640201 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:23.642333 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.643029 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.645109 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.645516 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.645629 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.649901 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:23.649907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:23.649910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:23.649951 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:23.652571 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.653665 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.653670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.653672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.653805 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.654350 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.654353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.654355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.654672 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:23.654799 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:23.655792 (+ 993us) server_negotiation.cc:299] Negotiation successful
0928 11:03:23.655984 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 11:03:23.657277 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.636894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:32783 (local address 127.0.0.1:41130)
0928 11:03:23.637738 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.637783 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:23.639716 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:23.640460 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:23.640472 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:23.641187 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:23.642101 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.642127 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.643186 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.643191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.644937 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.644954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.645742 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.645748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.646075 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.646810 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:23.646844 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:23.649580 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:23.652774 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.652785 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.652808 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.653503 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.653957 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.653962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.653966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.654241 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.654808 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:23.654817 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:23.655320 (+ 503us) client_negotiation.cc:769] Sending connection context
0928 11:03:23.655671 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 11:03:23.656111 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":336,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.124@KRBTEST.COM" created.
I20260928 11:03:23.659209 26630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:23 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:03:23.674690 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.659955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34902)
0928 11:03:23.660161 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.660179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:23.660290 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:23.661220 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:23.661225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:23.661684 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:23.661914 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.661926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.663272 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.663278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.664408 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.664424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.665256 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.665266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.665448 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.666104 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:23.666126 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:23.668557 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:23.672352 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.672356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.672359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.672733 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.673125 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.673129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.673131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.673273 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.673951 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:23.673955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:23.674079 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 11:03:23.674202 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 11:03:23.674416 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 11:03:23.675005 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.660134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.3.252.125:34717)
0928 11:03:23.660792 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:23.660798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:23.660826 (+ 28us) server_negotiation.cc:407] Connection header received
0928 11:03:23.660870 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:23.660876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:23.661071 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:23.661186 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:23.662074 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.663138 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.664552 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.665108 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.665201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.668782 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:23.668797 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:23.668815 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:23.668850 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:23.672207 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.672862 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.672871 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.672874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.673008 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.673374 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.673378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.673381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.673804 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:23.674028 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:23.674355 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 11:03:23.674608 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":330,"threads_started":1}
I20260928 11:03:23.677798 26630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:03:23.679005 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.678537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49750)
0928 11:03:23.678720 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.678852 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 11:03:23.679291 26579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
I20260928 11:03:23.680941 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.680545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49762)
0928 11:03:23.680723 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.680816 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 11:03:23.681190 26630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab.
Entry for principal kudu/127.3.252.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab.
I20260928 11:03:23.693153 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:36225
--webserver_interface=127.3.252.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab
--principal=kudu/127.3.252.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:32783,127.3.252.125:34717,127.3.252.124:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:03:23.730499 26630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 1
I20260928 11:03:23.731752 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.731225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49776)
0928 11:03:23.731448 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.731577 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 11:03:23.732807 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.732411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49782)
0928 11:03:23.732592 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.732671 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 11:03:23.733008 26630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
I20260928 11:03:23.801244 26630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 2
I20260928 11:03:23.802573 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.802022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49796)
0928 11:03:23.802244 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.802334 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 11:03:23.803774 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.803342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49800)
0928 11:03:23.803520 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.803616 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 11:03:23.804075 26630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
I20260928 11:03:23.808859 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } attempt: 5
Sep 28 11:03:23 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:03:23.821611 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.809494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34908)
0928 11:03:23.809677 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.809694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:23.809807 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:23.810229 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:23.810235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:23.810619 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:23.810803 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.810812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.811759 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.811763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.812603 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:23.812615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.813399 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.813405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:23.813612 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.814203 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:23.814221 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:23.816144 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:23.819195 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.819201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.819205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.819616 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.819988 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:23.819993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:23.819996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:23.820130 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:23.820655 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:23.820660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:23.820813 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 11:03:23.820935 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 11:03:23.821231 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 11:03:23.821612 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.809716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.3.252.125:34717)
0928 11:03:23.809921 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:23.809927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:23.809944 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:03:23.810003 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:23.810006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:23.810085 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:23.810166 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:23.810948 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.811629 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.812744 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:23.813237 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:23.813349 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:23.816337 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:23.816342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:23.816345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:23.816380 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:23.819043 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.819740 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.819744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.819746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.819872 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:23.820236 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:23.820240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:23.820242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:23.820541 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:23.820687 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:23.821057 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 11:03:23.821285 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 11:03:23.822793 26556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:03:23.823917 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.823547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49808)
0928 11:03:23.823701 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.823776 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 11:03:23.824774 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.824469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49810)
0928 11:03:23.824612 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.824681 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 11:03:23.824960 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
I20260928 11:03:23.876269 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 1
I20260928 11:03:23.877348 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.876890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49814)
0928 11:03:23.877084 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.877172 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 11:03:23.878309 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.877994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49818)
0928 11:03:23.878134 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.878203 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 11:03:23.878497 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
I20260928 11:03:23.895292 26630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 3
I20260928 11:03:23.896697 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.896039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49828)
0928 11:03:23.896340 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.896478 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 11:03:23.898013 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.897480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49834)
0928 11:03:23.897719 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.897821 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260928 11:03:23.898298 26630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
I20260928 11:03:23.946708 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 2
I20260928 11:03:23.947839 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.947362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49838)
0928 11:03:23.947561 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.947665 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 11:03:23.948740 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:23.948430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49846)
0928 11:03:23.948570 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:23.948637 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 11:03:23.948938 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
W20260928 11:03:24.024457 26637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 11:03:24.024847 26637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:24.024945 26637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:24.025064 26637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:24.032559 26637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:24.032671 26637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:24.032752 26637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:24.032817 26637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:24.032902 26637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:24.041544 26630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 4
I20260928 11:03:24.042799 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.042238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49860)
0928 11:03:24.042493 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.042601 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 11:03:24.043435 26637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:32783,127.3.252.125:34717,127.3.252.124:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab
--principal=kudu/127.3.252.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.252.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:24.043855 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.043495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49870)
0928 11:03:24.043645 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.043730 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 11:03:24.044073 26630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
I20260928 11:03:24.044863 26637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:24.046162 26637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 11:03:24.055137 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 3
I20260928 11:03:24.056238 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.055735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49880)
0928 11:03:24.055965 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.056048 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 11:03:24.057641 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.057151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49894)
0928 11:03:24.057402 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.057485 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 11:03:24.057582 26642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:24.057953 26556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:36225: Network error: Client connection negotiation failed: client connection to 127.3.252.124:36225: connect: Connection refused (error 111)
W20260928 11:03:24.057667 26645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:24.058058 26643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:24.059093 26637 server_base.cc:1034] running on GCE node
I20260928 11:03:24.059859 26637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:24.061302 26637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:24.062532 26637 hybrid_clock.cc:648] HybridClock initialized: now 1790593404062472 us; error 28 us; skew 500 ppm
I20260928 11:03:24.062899 26637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:24.066011 26637 init.cc:377] Logged in from keytab as kudu/127.3.252.124@KRBTEST.COM (short username kudu)
I20260928 11:03:24.067497 26637 webserver.cc:460] Webserver started at http://127.3.252.124:42625/ using document root <none> and password file <none>
I20260928 11:03:24.068256 26637 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:24.068356 26637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:24.068665 26637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:24.071689 26637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data/instance:
uuid: "0c79d4976c8548c4b1e6c0be9fbec808"
format_stamp: "Formatted at 2026-09-28 11:03:24 on dist-test-slave-djjw"
server_key: "6395ad0ba57d407291e6a69f164a4e27"
server_key_iv: "51291247abdce8a6d97456efc08b19d2"
server_key_version: "encryptionkey@0"
I20260928 11:03:24.072484 26637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/wal/instance:
uuid: "0c79d4976c8548c4b1e6c0be9fbec808"
format_stamp: "Formatted at 2026-09-28 11:03:24 on dist-test-slave-djjw"
server_key: "6395ad0ba57d407291e6a69f164a4e27"
server_key_iv: "51291247abdce8a6d97456efc08b19d2"
server_key_version: "encryptionkey@0"
I20260928 11:03:24.077703 26637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 11:03:24.081174 26652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:24.082518 26637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 11:03:24.082695 26637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/wal
uuid: "0c79d4976c8548c4b1e6c0be9fbec808"
format_stamp: "Formatted at 2026-09-28 11:03:24 on dist-test-slave-djjw"
server_key: "6395ad0ba57d407291e6a69f164a4e27"
server_key_iv: "51291247abdce8a6d97456efc08b19d2"
server_key_version: "encryptionkey@0"
I20260928 11:03:24.082901 26637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:24.099298 26637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:24.103644 26637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:24.103897 26637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:24.121699 26637 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.124:36225
I20260928 11:03:24.121709 26703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.124:36225 every 8 connection(s)
I20260928 11:03:24.123427 26637 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
I20260928 11:03:24.127607 26704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:24.129230 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26637
I20260928 11:03:24.129413 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790593056029081-4081-0/minicluster-data/master-2/wal/instance
I20260928 11:03:24.129997 4081 external_mini_cluster.cc:1428] Setting key 49bf87218f576a58bbcc8cb53c60640d
I20260928 11:03:24.137279 26704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:03:24.153882 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.131864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.3.252.124:36225)
0928 11:03:24.132631 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.132642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.132684 (+ 42us) server_negotiation.cc:407] Connection header received
0928 11:03:24.134276 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.134301 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.134709 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.135034 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.136694 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.138009 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.139739 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.140781 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.140954 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.145492 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.145503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.145518 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.145566 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.149015 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.149932 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.149941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.149951 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.150166 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.150555 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.150560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.150563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.151283 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.151639 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.152039 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.152522 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
I20260928 11:03:24.155858 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.140638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.3.252.126:32783)
0928 11:03:24.140864 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.140870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.141196 (+ 326us) server_negotiation.cc:407] Connection header received
0928 11:03:24.141371 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.141375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.141458 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.141556 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.142970 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.143679 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.145316 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.145988 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.146107 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.149967 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.149973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.149977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.150020 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.153176 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.153879 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.153884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.153887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.154038 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.154461 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.154466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.154470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.154728 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.154851 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.155283 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.155497 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 11:03:24.155900 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.140246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:32783 (local address 127.0.0.1:41138)
0928 11:03:24.141040 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.141091 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:24.141256 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:24.141614 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:24.141620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:24.142492 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:24.142824 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.142837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.143836 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.143841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.145169 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.145184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.146194 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.146199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.146410 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.147029 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:24.147052 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:24.149716 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:24.153308 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.153317 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.153323 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.153728 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.154181 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.154186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.154189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.154352 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.154855 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:24.154864 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:24.155019 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 11:03:24.155161 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 11:03:24.155495 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":460,"threads_started":1}
I20260928 11:03:24.157153 26704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:03:24.173905 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.156856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.3.252.126:32783)
0928 11:03:24.157012 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.157016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.157030 (+ 14us) server_negotiation.cc:407] Connection header received
0928 11:03:24.157161 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.157164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.157222 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.157294 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.159022 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.160251 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.162225 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.162913 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.163104 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.168241 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.168247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.168251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.168293 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.171137 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.171926 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.171932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.171934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.172080 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.172454 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.172458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.172460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.172759 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.173026 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.173328 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.173565 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 11:03:24.173906 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.157950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:34717 (local address 127.0.0.1:34924)
0928 11:03:24.158115 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.158140 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:24.158249 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:24.158542 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:24.158546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:24.158961 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:24.159133 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.159142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.160552 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.160558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.161997 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.162017 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.163128 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.163135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.163317 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.164117 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:24.164143 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:24.168142 (+ 3999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:24.171331 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.171337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.171341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.171775 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.172208 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.172213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.172216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.172379 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.172902 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:24.172907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:24.173067 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 11:03:24.173187 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 11:03:24.173506 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 11:03:24.173926 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.157957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.3.252.125:34717)
0928 11:03:24.158125 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.158131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.158287 (+ 156us) server_negotiation.cc:407] Connection header received
0928 11:03:24.158350 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.158354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.158417 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.158500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.159282 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.160357 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.162225 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.162913 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.163063 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.168440 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.168446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.168449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.168489 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.171148 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.171935 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.171939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.171942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.172088 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.172493 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.172497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.172500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.172777 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.172916 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.173322 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.173534 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 11:03:24.175365 26704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:03:24.189080 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.176159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49916)
0928 11:03:24.176374 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.176392 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:24.176481 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:24.176967 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:24.176970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:24.177433 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:24.177567 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.177575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.178549 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.178554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.179378 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.179388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.180055 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.180061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.180237 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.180888 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:24.180905 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:24.183302 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:24.186511 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.186517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.186521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.186909 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.187321 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.187326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.187329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.187498 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.188048 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:24.188053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:24.188222 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 11:03:24.188355 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 11:03:24.188652 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 11:03:24.189116 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.176358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.3.252.124:36225)
0928 11:03:24.176482 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.176487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.176502 (+ 15us) server_negotiation.cc:407] Connection header received
0928 11:03:24.176634 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.176639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.176696 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.176779 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.177707 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.178432 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.179499 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.179903 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.180006 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.183490 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.183495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.183498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.183528 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.186367 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.187016 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.187021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.187024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.187167 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.187599 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.187603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.187606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.187929 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.188039 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.188457 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.188810 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":36}
I20260928 11:03:24.190207 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.175707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.3.252.125:34717)
0928 11:03:24.175868 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.175873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.175891 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:03:24.176036 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.176041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.176107 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.176181 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.177433 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.178208 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.179444 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.179880 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.179992 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.184620 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.184625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.184629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.184673 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.187418 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.188176 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.188180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.188182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.188320 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.188717 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.188720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.188722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.188979 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.189256 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.189641 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.189883 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 11:03:24.194967 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:24.195175 26704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: Bootstrap starting.
I20260928 11:03:24.199118 26704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:24.200194 26704 log.cc:824] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:24.198581 26531 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "069c9505661842ec87e0b2753d0c1c00"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41152
I20260928 11:03:24.203336 26704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: No bootstrap required, opened a new log
I20260928 11:03:24.210700 26704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.211056 26704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c79d4976c8548c4b1e6c0be9fbec808, State: Initialized, Role: FOLLOWER
I20260928 11:03:24.211478 26704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.212419 26710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.212826 26710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:24.213380 26704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 11:03:24.222669 26721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 11:03:24.222834 26721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 11:03:24.220701 26637 master_runner.cc:186] Error getting master registration for 127.3.252.126:32783: OK, instance_id { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" instance_seqno: 1790593403100845 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 11:03:24.229195 26637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 11:03:24.235177 26556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 4
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
I20260928 11:03:24.251320 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.236058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.3.252.124:36225)
0928 11:03:24.236293 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.236299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.236326 (+ 27us) server_negotiation.cc:407] Connection header received
0928 11:03:24.236493 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.236498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.236578 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.236696 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.237613 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.238825 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.240627 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.241335 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.241466 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.245103 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.245108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.245111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.245148 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.247953 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.248996 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.249002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.249004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.249159 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.249614 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.249618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.249621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.250067 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.250238 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.250704 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.250924 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 11:03:24.251305 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.235922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49932)
0928 11:03:24.236230 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.236250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:24.236370 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:24.236733 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:24.236738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:24.237248 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:24.237455 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.237467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.238968 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.238974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.240459 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.240474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.241479 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.241484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.241700 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.242420 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:24.242445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:24.244825 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:24.248121 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.248128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.248133 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.248730 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.249286 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.249291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.249295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.249475 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.250212 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:24.250217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:24.250424 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 11:03:24.250573 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 11:03:24.250895 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 11:03:24.255985 26556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00: Bootstrap starting.
I20260928 11:03:24.260053 26556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:24.261492 26556 log.cc:824] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:24.264518 26556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00: No bootstrap required, opened a new log
I20260928 11:03:24.269728 26556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.270133 26556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069c9505661842ec87e0b2753d0c1c00, State: Initialized, Role: FOLLOWER
I20260928 11:03:24.270576 26556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.271602 26724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.271986 26724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:24.272804 26556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:24.272910 26531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 11:03:24.273108 26531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:24.276717 26531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.279793 26679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "069c9505661842ec87e0b2753d0c1c00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c79d4976c8548c4b1e6c0be9fbec808"
I20260928 11:03:24.280493 26679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:24.284794 26679 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069c9505661842ec87e0b2753d0c1c00 in term 1.
I20260928 11:03:24.285677 26505 leader_election.cc:302] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069c9505661842ec87e0b2753d0c1c00, 0c79d4976c8548c4b1e6c0be9fbec808; no voters:
I20260928 11:03:24.286237 26724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:24.286584 26531 leader_election.cc:288] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [CANDIDATE]: Term 1 election: Requested vote from peers 89fb661b65f7431d90249ca5a293610d (127.3.252.125:34717), 0c79d4976c8548c4b1e6c0be9fbec808 (127.3.252.124:36225)
I20260928 11:03:24.300936 26724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [term 1 LEADER]: Becoming Leader. State: Replica: 069c9505661842ec87e0b2753d0c1c00, State: Running, Role: LEADER
I20260928 11:03:24.302211 26724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.316387 26731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069c9505661842ec87e0b2753d0c1c00. Latest consensus state: current_term: 1 leader_uuid: "069c9505661842ec87e0b2753d0c1c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.316915 26731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:24.323031 26741 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 11:03:24.329797 26742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:24.327982 26489 master_runner.cc:186] Error getting master registration for 127.3.252.125:34717: OK, instance_id { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" instance_seqno: 1790593403614419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 11:03:24.341457 26510 tablet.cc:2369] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:03:24.343291 26630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } attempt: 5
I20260928 11:03:24.344261 26489 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 11:03:24.346871 26742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 11:03:24.356029 26658 tablet.cc:2369] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:03:24.370208 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.327711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.3.252.124:36225)
0928 11:03:24.327875 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.327880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.346294 (+ 18414us) server_negotiation.cc:407] Connection header received
0928 11:03:24.346829 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.346837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.346935 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.347006 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.349181 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.350432 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.353218 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.353926 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.354061 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.360270 (+ 6209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.360277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.360282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.360338 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.364191 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.365139 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.365145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.365149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.365323 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.366274 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.366280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.366283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.366737 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.367127 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.369496 (+ 2369us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.369822 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 11:03:24.378942 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.320653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.3.252.126:32783)
0928 11:03:24.320858 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.320865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.342587 (+ 21722us) server_negotiation.cc:407] Connection header received
0928 11:03:24.343158 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.343165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.343257 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.343339 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.358428 (+ 15089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.359537 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.362393 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.363133 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.363241 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.368529 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.368536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.368545 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.368593 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.372597 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.375133 (+ 2536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.375140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.375144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.375370 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.376097 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.376103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.376106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.376528 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.376811 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.377799 (+ 988us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.378287 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 11:03:24.380870 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.326918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.3.252.125:34717)
0928 11:03:24.327560 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.327566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.353097 (+ 25531us) server_negotiation.cc:407] Connection header received
0928 11:03:24.353417 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.353423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.353519 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.353600 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.355600 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.356660 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.367740 (+ 11080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.368437 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.368538 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.373290 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.373298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.373303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.373355 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.376808 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.377606 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.377617 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.377621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.377807 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.378254 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.378259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.378261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.378643 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.378876 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.379769 (+ 893us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.380496 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514}
Sep 28 11:03:24 dist-test-slave-djjw krb5kdc[26476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
I20260928 11:03:24.388248 26679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [term 1 FOLLOWER]: Refusing update from remote peer 069c9505661842ec87e0b2753d0c1c00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:03:24.390199 26731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:03:24.391731 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.343910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:36225 (local address 127.0.0.1:49946)
0928 11:03:24.344299 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:24.344319 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:24.344428 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:24.370002 (+ 25574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:24.370008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:24.370523 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:24.370759 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.370774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.377952 (+ 7178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.377958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.379414 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:24.379430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.380484 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.380491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:24.380767 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.381417 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:24.381441 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:24.384193 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:24.388650 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.388655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.388659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.389230 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.389687 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:24.389692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:24.389695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:24.389883 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:24.390737 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:24.390741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:24.390896 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 11:03:24.391061 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 11:03:24.391373 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260928 11:03:24.398845 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.344267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.3.252.124:36225)
0928 11:03:24.363172 (+ 18905us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.363179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.363207 (+ 28us) server_negotiation.cc:407] Connection header received
0928 11:03:24.363275 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.363281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.363370 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.371030 (+ 7660us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.373809 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.375152 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.379639 (+ 4487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.380298 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.382994 (+ 2696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.384422 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:24.384428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:24.384434 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:24.384476 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:24.388469 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.389394 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.389398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.389402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.389555 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:24.390042 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:24.390046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:24.390050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:24.390609 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:24.390747 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.397053 (+ 6306us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.398364 (+ 1311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18725,"thread_start_us":19215,"threads_started":1}
I20260928 11:03:24.406077 26630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d: Bootstrap starting.
I20260928 11:03:24.408301 26710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069c9505661842ec87e0b2753d0c1c00. Latest consensus state: current_term: 1 leader_uuid: "069c9505661842ec87e0b2753d0c1c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.408605 26710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:24.410281 26630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:24.411666 26630 log.cc:824] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:24.425091 26630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d: No bootstrap required, opened a new log
I20260928 11:03:24.431782 26731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069c9505661842ec87e0b2753d0c1c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.432080 26731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:24.433302 26742 catalog_manager.cc:1347] Generated new cluster ID: 770210cf459e489c9535b10bbe780cb7
I20260928 11:03:24.433408 26742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:24.434449 26710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069c9505661842ec87e0b2753d0c1c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.434721 26710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:24.448742 26630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.449523 26630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fb661b65f7431d90249ca5a293610d, State: Initialized, Role: FOLLOWER
I20260928 11:03:24.450505 26630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } }
I20260928 11:03:24.463485 26630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:24.463896 26604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:24.464529 26742 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:24.466300 26605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "069c9505661842ec87e0b2753d0c1c00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "89fb661b65f7431d90249ca5a293610d"
I20260928 11:03:24.469373 26742 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:24.470775 26604 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [term 1 FOLLOWER]: Refusing update from remote peer 069c9505661842ec87e0b2753d0c1c00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:03:24.471995 26754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.472414 26754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:24.474908 26605 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069c9505661842ec87e0b2753d0c1c00 in term 1.
I20260928 11:03:24.479548 26724 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:03:24.493317 26742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00: Generated new TSK 0
I20260928 11:03:24.494529 26742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 11:03:24.534525 26754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069c9505661842ec87e0b2753d0c1c00. Latest consensus state: current_term: 1 leader_uuid: "069c9505661842ec87e0b2753d0c1c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.534898 26754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:24.534874 26760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069c9505661842ec87e0b2753d0c1c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.535259 26760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:24.536451 26731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069c9505661842ec87e0b2753d0c1c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c9505661842ec87e0b2753d0c1c00" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 32783 } } peers { permanent_uuid: "89fb661b65f7431d90249ca5a293610d" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 34717 } } peers { permanent_uuid: "0c79d4976c8548c4b1e6c0be9fbec808" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 36225 } } }
I20260928 11:03:24.536772 26731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c9505661842ec87e0b2753d0c1c00 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:24.655434 26773 catalog_manager.cc:1259] Loaded cluster ID: 770210cf459e489c9535b10bbe780cb7
I20260928 11:03:24.655560 26773 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d: loading cluster ID for follower catalog manager: success
I20260928 11:03:24.659902 26773 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d: acquiring CA information for follower catalog manager: success
I20260928 11:03:24.671571 26773 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89fb661b65f7431d90249ca5a293610d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 11:03:24.678265 26559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.654628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.3.252.126:32783)
0928 11:03:24.654848 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.654855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.675216 (+ 20361us) server_negotiation.cc:407] Connection header received
0928 11:03:24.675457 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.675463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.675558 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.675666 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.678015 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41176: BlockingRecv error: recv got EOF from 127.0.0.1:41176 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 11:03:24.681010 26634 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.661495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.3.252.125:34717)
0928 11:03:24.661684 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.661691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.678822 (+ 17131us) server_negotiation.cc:407] Connection header received
0928 11:03:24.679514 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.679520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.679615 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.679688 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.680767 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34946: BlockingRecv error: recv got EOF from 127.0.0.1:34946 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 11:03:24.683496 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.661626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.3.252.124:36225)
0928 11:03:24.661861 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.661867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.681881 (+ 20014us) server_negotiation.cc:407] Connection header received
0928 11:03:24.682032 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.682038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.682122 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.682214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.683265 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49956: BlockingRecv error: recv got EOF from 127.0.0.1:49956 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 11:03:24.739048 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.718892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.3.252.125:34717)
0928 11:03:24.719097 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.719103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.730542 (+ 11439us) server_negotiation.cc:407] Connection header received
0928 11:03:24.730855 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.730862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.731110 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.731267 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:24.732102 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.733318 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.735526 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.736212 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.736317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.737453 (+ 1136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:24.738183 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:24.738308 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.738575 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.738680 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 11:03:24.786556 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.773074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.3.252.124:36225)
0928 11:03:24.773285 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.773291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.784165 (+ 10874us) server_negotiation.cc:407] Connection header received
0928 11:03:24.784770 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.784777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.784923 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.785019 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.786255 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49972: BlockingRecv error: recv got EOF from 127.0.0.1:49972 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:03:24.838073 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.822748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.3.252.125:34717)
0928 11:03:24.822946 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.822953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.830910 (+ 7957us) server_negotiation.cc:407] Connection header received
0928 11:03:24.831463 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.831470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.831602 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.831667 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:24.832391 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.833569 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.835681 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.836416 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.836518 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.837160 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:24.837313 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:24.837386 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.837608 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.837707 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 11:03:24.886206 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.873043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.3.252.124:36225)
0928 11:03:24.873242 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.873248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.884044 (+ 10796us) server_negotiation.cc:407] Connection header received
0928 11:03:24.884568 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.884575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.884682 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.884756 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.885905 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49982: BlockingRecv error: recv got EOF from 127.0.0.1:49982 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 11:03:24.921046 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.909976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.3.252.125:34717)
0928 11:03:24.910169 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.910176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.914276 (+ 4100us) server_negotiation.cc:407] Connection header received
0928 11:03:24.914595 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.914602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.914721 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.914793 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:24.915479 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.916568 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.918778 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.919544 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.919693 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.920191 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:24.920342 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:24.920436 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.920603 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.920697 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 11:03:24.931432 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.929539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.3.252.124:36225)
0928 11:03:24.929745 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.929752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.929775 (+ 23us) server_negotiation.cc:407] Connection header received
0928 11:03:24.930120 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.930127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.930232 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.930346 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.931186 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49984: BlockingRecv error: recv got EOF from 127.0.0.1:49984 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 11:03:24.948836 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.939599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.3.252.125:34717)
0928 11:03:24.939848 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.939855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.942438 (+ 2583us) server_negotiation.cc:407] Connection header received
0928 11:03:24.942629 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.942636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.942735 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.942845 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:24.943317 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.944456 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.946538 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.947311 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.947442 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.947964 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:24.948103 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:24.948194 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.948384 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.948469 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260928 11:03:24.956722 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.954645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.3.252.124:36225)
0928 11:03:24.954887 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.954893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.955292 (+ 399us) server_negotiation.cc:407] Connection header received
0928 11:03:24.955472 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.955476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.955563 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.955644 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.956490 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49988: BlockingRecv error: recv got EOF from 127.0.0.1:49988 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 11:03:24.974500 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:24.968886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.3.252.125:34717)
0928 11:03:24.969083 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.969090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.969335 (+ 245us) server_negotiation.cc:407] Connection header received
0928 11:03:24.969502 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.969507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.969602 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.969703 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:24.970096 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.971170 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.972532 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:24.973222 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:24.973335 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:24.973720 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:24.973902 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:24.973985 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:24.974103 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 11:03:24.974185 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 11:03:24.982726 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:24.980759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.3.252.124:36225)
0928 11:03:24.980959 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:24.980964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:24.981311 (+ 347us) server_negotiation.cc:407] Connection header received
0928 11:03:24.981486 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:24.981490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:24.981567 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:24.981649 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:24.982495 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49990: BlockingRecv error: recv got EOF from 127.0.0.1:49990 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 11:03:25.008723 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.002694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.3.252.125:34717)
0928 11:03:25.002899 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.002905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.003190 (+ 285us) server_negotiation.cc:407] Connection header received
0928 11:03:25.003366 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.003372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.003470 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.003598 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.003983 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.005175 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.006662 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.007329 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.007459 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.007894 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.008084 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.008184 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.008337 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.008429 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 11:03:25.017300 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:25.015301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.3.252.124:36225)
0928 11:03:25.015488 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.015495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.015809 (+ 314us) server_negotiation.cc:407] Connection header received
0928 11:03:25.016047 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.016054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.016151 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.016266 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:25.017057 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49994: BlockingRecv error: recv got EOF from 127.0.0.1:49994 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 11:03:25.059264 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.053153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.3.252.125:34717)
0928 11:03:25.053389 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.053396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.053726 (+ 330us) server_negotiation.cc:407] Connection header received
0928 11:03:25.053925 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.053931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.054064 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.054183 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.054635 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.055760 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.057145 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.057903 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.058044 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.058449 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.058616 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.058715 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.058835 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.058924 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 11:03:25.067147 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:25.065226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.3.252.124:36225)
0928 11:03:25.065438 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.065444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.065804 (+ 360us) server_negotiation.cc:407] Connection header received
0928 11:03:25.065997 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.066004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.066114 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.066232 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:25.066922 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50008: BlockingRecv error: recv got EOF from 127.0.0.1:50008 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 11:03:25.145473 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.139298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.3.252.125:34717)
0928 11:03:25.139472 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.139477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.139997 (+ 520us) server_negotiation.cc:407] Connection header received
0928 11:03:25.140176 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.140182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.140270 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.140365 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.140840 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.142162 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.143543 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.144198 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.144319 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.144721 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.144865 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.144944 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.145088 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.145172 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 11:03:25.153227 26747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:25.151291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.3.252.124:36225)
0928 11:03:25.151498 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.151516 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.151753 (+ 237us) server_negotiation.cc:407] Connection header received
0928 11:03:25.151917 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.151921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.151995 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.152079 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:25.152937 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50012: BlockingRecv error: recv got EOF from 127.0.0.1:50012 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 11:03:25.224836 26721 catalog_manager.cc:1259] Loaded cluster ID: 770210cf459e489c9535b10bbe780cb7
I20260928 11:03:25.224936 26721 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: loading cluster ID for follower catalog manager: success
I20260928 11:03:25.227368 26721 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: acquiring CA information for follower catalog manager: success
I20260928 11:03:25.229146 26721 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c79d4976c8548c4b1e6c0be9fbec808: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 11:03:25.290058 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.285258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.3.252.125:34717)
0928 11:03:25.285482 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.285488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.285687 (+ 199us) server_negotiation.cc:407] Connection header received
0928 11:03:25.285882 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.285886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.285976 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.286066 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.286513 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.287237 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.288619 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.289010 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.289102 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.289426 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.289539 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.289600 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.289693 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.289755 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 11:03:25.300714 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.295482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.3.252.124:36225)
0928 11:03:25.295708 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.295715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.295950 (+ 235us) server_negotiation.cc:407] Connection header received
0928 11:03:25.296090 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.296094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.296253 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.296351 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.296641 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.297423 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.298732 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.299155 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.299255 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.299684 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.300210 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.300316 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.300357 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.300422 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 11:03:25.324251 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.307297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.3.252.125:34717)
0928 11:03:25.307667 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.307674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.318415 (+ 10741us) server_negotiation.cc:407] Connection header received
0928 11:03:25.318582 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.318586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.318671 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.318760 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.319601 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.320723 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.322353 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.322786 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.322899 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.323488 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.323635 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.323696 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.323880 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.323964 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260928 11:03:25.325189 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.307197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.3.252.124:36225)
0928 11:03:25.307492 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.307499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.319605 (+ 12106us) server_negotiation.cc:407] Connection header received
0928 11:03:25.319762 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.319766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.319842 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.319917 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.320367 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.321222 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.323203 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.323739 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.323849 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.324529 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.324639 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.324711 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.324846 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.324925 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260928 11:03:25.325932 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:25.306912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.3.252.126:32783)
0928 11:03:25.317994 (+ 11082us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:25.318002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:25.318037 (+ 35us) server_negotiation.cc:407] Connection header received
0928 11:03:25.318183 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:25.318189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:25.318430 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:25.318645 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:25.319802 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.320826 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.322876 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:25.323480 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:25.323585 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:25.324887 (+ 1302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:25.325334 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:25.325437 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:25.325564 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 11:03:25.325647 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10927,"thread_start_us":11867,"threads_started":1}
I20260928 11:03:25.331002 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26489
I20260928 11:03:25.332085 26551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:25.441429 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26489
W20260928 11:03:25.458237 26582 connection.cc:535] server connection from 127.0.0.1:34908 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 11:03:25.458606 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26563
I20260928 11:03:25.459676 26625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:25.565601 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26563
I20260928 11:03:25.582152 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26637
I20260928 11:03:25.583189 26699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:25.689244 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26637
2026-09-28T11:03:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4296 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 11:03:25 dist-test-slave-djjw krb5kdc[26884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 11:03:25 dist-test-slave-djjw krb5kdc[26884](info): set up 2 sockets
Sep 28 11:03:25 dist-test-slave-djjw krb5kdc[26884](info): commencing operation
krb5kdc: starting...
W20260928 11:03:26.719084 4081 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/test-user.keytab.
Sep 28 11:03:26 dist-test-slave-djjw krb5kdc[26884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T11:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T11:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.252.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
Entry for principal kudu/127.3.252.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab.
I20260928 11:03:26.980922 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36661
--webserver_interface=127.3.252.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36661,127.3.252.125:39695,127.3.252.124:39997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 11:03:27.330703 26897 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 11:03:27.331063 26897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:27.331144 26897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:27.331285 26897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:27.338819 26897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:27.338923 26897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:27.338984 26897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:27.339053 26897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:27.339130 26897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:27.349411 26897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36661,127.3.252.125:39695,127.3.252.124:39997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.126.keytab
--principal=kudu/127.3.252.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.126:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.252.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:27.350811 26897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:27.352095 26897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:27.363659 26902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:27.363591 26903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:27.363622 26905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:27.364442 26897 server_base.cc:1034] running on GCE node
I20260928 11:03:27.365275 26897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:27.366824 26897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:27.368120 26897 hybrid_clock.cc:648] HybridClock initialized: now 1790593407368034 us; error 58 us; skew 500 ppm
I20260928 11:03:27.368461 26897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:27 dist-test-slave-djjw krb5kdc[26884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:27.371587 26897 init.cc:377] Logged in from keytab as kudu/127.3.252.126@KRBTEST.COM (short username kudu)
I20260928 11:03:27.373158 26897 webserver.cc:460] Webserver started at http://127.3.252.126:42495/ using document root <none> and password file <none>
I20260928 11:03:27.373931 26897 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:27.374047 26897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:27.374356 26897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:27.377458 26897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data/instance:
uuid: "95ffe625d6ef41dd8d0f5a898a5afca5"
format_stamp: "Formatted at 2026-09-28 11:03:27 on dist-test-slave-djjw"
server_key: "86e475b02208861a49ed907f91a28e8e"
server_key_iv: "55b1a4c11b65572adbdfedc2a3577068"
server_key_version: "encryptionkey@0"
I20260928 11:03:27.378309 26897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/wal/instance:
uuid: "95ffe625d6ef41dd8d0f5a898a5afca5"
format_stamp: "Formatted at 2026-09-28 11:03:27 on dist-test-slave-djjw"
server_key: "86e475b02208861a49ed907f91a28e8e"
server_key_iv: "55b1a4c11b65572adbdfedc2a3577068"
server_key_version: "encryptionkey@0"
I20260928 11:03:27.383695 26897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 11:03:27.387337 26912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:27.388895 26897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:03:27.389132 26897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/wal
uuid: "95ffe625d6ef41dd8d0f5a898a5afca5"
format_stamp: "Formatted at 2026-09-28 11:03:27 on dist-test-slave-djjw"
server_key: "86e475b02208861a49ed907f91a28e8e"
server_key_iv: "55b1a4c11b65572adbdfedc2a3577068"
server_key_version: "encryptionkey@0"
I20260928 11:03:27.389367 26897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:27.402384 26897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:27.406818 26897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:27.407090 26897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:27.424638 26897 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.126:36661
I20260928 11:03:27.424656 26963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.126:36661 every 8 connection(s)
I20260928 11:03:27.426083 26897 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/data/info.pb
I20260928 11:03:27.429800 26964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:27.435935 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26897
I20260928 11:03:27.436095 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-0/wal/instance
I20260928 11:03:27.436512 4081 external_mini_cluster.cc:1428] Setting key acce5f9a0822ac3063c7ba55bb88a4a4
I20260928 11:03:27.436467 26964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:27 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.125@KRBTEST.COM; defaulting to no policy
I20260928 11:03:27.458911 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.439828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36661 (local address 127.0.0.1:56282)
0928 11:03:27.440951 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.440986 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:27.442799 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:27.443687 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:27.443695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:27.444406 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:27.444761 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:27.444780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.446391 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.446395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:27.447479 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:27.447491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.448410 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.448417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:27.448673 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:27.449288 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:27.449315 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:27.451520 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:27.454658 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:27.454669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:27.454683 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:27.455152 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:27.455532 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:27.455537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:27.455541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:27.455727 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:27.456528 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:27.456536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:27.456737 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 11:03:27.456960 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 11:03:27.457551 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":194,"thread_start_us":700,"threads_started":1}
I20260928 11:03:27.458943 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.439827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.3.252.126:36661)
0928 11:03:27.440658 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:27.440676 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:27.441123 (+ 447us) server_negotiation.cc:407] Connection header received
0928 11:03:27.443189 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:27.443195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:27.443446 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:27.443630 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:27.444965 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.446234 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.447610 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.448187 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.448358 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:27.451768 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:27.451777 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:27.451790 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:27.451820 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:27.454529 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:27.455274 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:27.455281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:27.455283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:27.455428 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:27.455832 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:27.455835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:27.455838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:27.456392 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:27.456593 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:27.457108 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 11:03:27.457412 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":568,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.125@KRBTEST.COM" created.
I20260928 11:03:27.461954 26964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:03:27.463052 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.462604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48220)
0928 11:03:27.462790 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.462918 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 11:03:27.463328 26913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
I20260928 11:03:27.464810 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.464429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48236)
0928 11:03:27.464576 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.464661 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 11:03:27.465000 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:39695: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab.
Entry for principal kudu/127.3.252.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab.
I20260928 11:03:27.492573 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:39695
--webserver_interface=127.3.252.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab
--principal=kudu/127.3.252.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36661,127.3.252.125:39695,127.3.252.124:39997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:03:27.514351 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } attempt: 1
I20260928 11:03:27.515568 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.515085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48238)
0928 11:03:27.515309 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.515431 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 11:03:27.516665 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.516263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48246)
0928 11:03:27.516465 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.516536 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 11:03:27.516855 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:39695: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
I20260928 11:03:27.585099 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } attempt: 2
I20260928 11:03:27.586216 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.585752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48248)
0928 11:03:27.585982 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.586082 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 11:03:27.587230 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.586864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48264)
0928 11:03:27.587012 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.587108 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 11:03:27.587409 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:39695: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
I20260928 11:03:27.678629 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } attempt: 3
I20260928 11:03:27.679687 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.679259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48268)
0928 11:03:27.679455 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.679550 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 11:03:27.680725 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.680364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48276)
0928 11:03:27.680523 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.680592 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 11:03:27.680902 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:39695: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
W20260928 11:03:27.822973 26971 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 11:03:27.823338 26971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:27.823432 26971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:27.823541 26971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 11:03:27.824129 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } attempt: 4
I20260928 11:03:27.825243 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.824724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48286)
0928 11:03:27.825004 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.825092 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 11:03:27.826256 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.825920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48300)
0928 11:03:27.826054 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.826139 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 11:03:27.826416 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.125:39695: Network error: Client connection negotiation failed: client connection to 127.3.252.125:39695: connect: Connection refused (error 111)
W20260928 11:03:27.831257 26971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:27.831363 26971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:27.831455 26971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:27.831517 26971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:27.831600 26971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:27.841650 26971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36661,127.3.252.125:39695,127.3.252.124:39997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.125.keytab
--principal=kudu/127.3.252.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.125:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.252.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:27.843001 26971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:27.844350 26971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:27.855892 26977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:27.855834 26979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:27.856419 26976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:27.856822 26971 server_base.cc:1034] running on GCE node
I20260928 11:03:27.857581 26971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:27.859004 26971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:27.860235 26971 hybrid_clock.cc:648] HybridClock initialized: now 1790593407860164 us; error 39 us; skew 500 ppm
I20260928 11:03:27.860567 26971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:27 dist-test-slave-djjw krb5kdc[26884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:27.863817 26971 init.cc:377] Logged in from keytab as kudu/127.3.252.125@KRBTEST.COM (short username kudu)
I20260928 11:03:27.865199 26971 webserver.cc:460] Webserver started at http://127.3.252.125:34233/ using document root <none> and password file <none>
I20260928 11:03:27.865885 26971 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:27.865976 26971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:27.866310 26971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:27.869254 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data/instance:
uuid: "0c41212e1891409e8ed50a85d6221ca2"
format_stamp: "Formatted at 2026-09-28 11:03:27 on dist-test-slave-djjw"
server_key: "e2010c8068928bebf9298fcc3ebc53c3"
server_key_iv: "bbb5386351b06c5853a5b1974bc974c6"
server_key_version: "encryptionkey@0"
I20260928 11:03:27.870095 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/wal/instance:
uuid: "0c41212e1891409e8ed50a85d6221ca2"
format_stamp: "Formatted at 2026-09-28 11:03:27 on dist-test-slave-djjw"
server_key: "e2010c8068928bebf9298fcc3ebc53c3"
server_key_iv: "bbb5386351b06c5853a5b1974bc974c6"
server_key_version: "encryptionkey@0"
I20260928 11:03:27.875733 26971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 11:03:27.879386 26986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:27.880836 26971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 11:03:27.880992 26971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/wal
uuid: "0c41212e1891409e8ed50a85d6221ca2"
format_stamp: "Formatted at 2026-09-28 11:03:27 on dist-test-slave-djjw"
server_key: "e2010c8068928bebf9298fcc3ebc53c3"
server_key_iv: "bbb5386351b06c5853a5b1974bc974c6"
server_key_version: "encryptionkey@0"
I20260928 11:03:27.881191 26971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:27.900802 26971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:27.905292 26971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:27.905553 26971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:27.921933 26971 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.125:39695
I20260928 11:03:27.921952 27037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.125:39695 every 8 connection(s)
I20260928 11:03:27.923324 26971 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/data/info.pb
I20260928 11:03:27.927202 27038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:27.927559 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 26971
I20260928 11:03:27.927745 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-1/wal/instance
I20260928 11:03:27.928283 4081 external_mini_cluster.cc:1428] Setting key c82b26aa42b8a1c1d303a5e6149679e9
I20260928 11:03:27.935981 27038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:27 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.252.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.252.124@KRBTEST.COM" created.
I20260928 11:03:27.956279 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.939233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.3.252.126:36661)
0928 11:03:27.939408 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:27.939412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:27.940733 (+ 1321us) server_negotiation.cc:407] Connection header received
0928 11:03:27.942683 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:27.942689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:27.942781 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:27.942865 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:27.944403 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.945078 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.946611 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.947015 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.947124 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:27.950604 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:27.950609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:27.950612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:27.950646 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:27.953183 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:27.953915 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:27.953920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:27.953923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:27.954062 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:27.954455 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:27.954459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:27.954461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:27.954778 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:27.954910 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:27.955709 (+ 799us) server_negotiation.cc:299] Negotiation successful
0928 11:03:27.955948 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:03:27.957127 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.939791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36661 (local address 127.0.0.1:56294)
0928 11:03:27.940577 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.940613 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:27.942395 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:27.943111 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:27.943119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:27.943653 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:27.944228 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:27.944246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.945237 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.945241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:27.946466 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:27.946478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.947246 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.947254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:27.947464 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:27.948094 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:27.948117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:27.950371 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:27.953320 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:27.953330 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:27.953349 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:27.953777 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:27.954161 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:27.954166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:27.954170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:27.954343 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:27.954904 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:27.954912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:27.955279 (+ 367us) client_negotiation.cc:769] Sending connection context
0928 11:03:27.955569 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 11:03:27.956009 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":352,"threads_started":1}
I20260928 11:03:27.958904 27038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:27 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:03:27.972455 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.959656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48312)
0928 11:03:27.959813 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.959827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:27.959913 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:27.960890 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:27.960894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:27.961315 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:27.961443 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:27.961452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.962818 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.962821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:27.963645 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:27.963655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.964435 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.964440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:27.964596 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:27.965147 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:27.965163 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:27.967141 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:27.970137 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:27.970143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:27.970147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:27.970475 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:27.970803 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:27.970808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:27.970811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:27.970935 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:27.971535 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:27.971540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:27.971679 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 11:03:27.971792 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 11:03:27.972061 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 11:03:27.972498 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.959911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.3.252.125:39695)
0928 11:03:27.960488 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:27.960494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:27.960518 (+ 24us) server_negotiation.cc:407] Connection header received
0928 11:03:27.960560 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:27.960565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:27.960749 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:27.960889 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:27.961594 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.962684 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.963774 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:27.964304 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:27.964409 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:27.967337 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:27.967349 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:27.967363 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:27.967391 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:27.970020 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:27.970589 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:27.970595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:27.970597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:27.970708 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:27.971023 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:27.971028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:27.971031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:27.971403 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:27.971582 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:27.971950 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 11:03:27.972157 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":321,"threads_started":1}
I20260928 11:03:27.975189 27038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:03:27.976461 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.975998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47448)
0928 11:03:27.976180 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.976304 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 11:03:27.976738 26989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:27.978346 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:27.978009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47462)
0928 11:03:27.978147 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:27.978232 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 11:03:27.978555 27038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.252.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab.
Entry for principal kudu/127.3.252.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab.
I20260928 11:03:27.987087 4081 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
/tmp/dist-test-tasktiZmwa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:39997
--webserver_interface=127.3.252.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab
--principal=kudu/127.3.252.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.252.84:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.252.126:36661,127.3.252.125:39695,127.3.252.124:39997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 11:03:28.027891 27038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 1
I20260928 11:03:28.029057 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.028605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47476)
0928 11:03:28.028803 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.028907 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 11:03:28.030045 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.029713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47478)
0928 11:03:28.029851 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.029942 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 11:03:28.030264 27038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:28.098498 27038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 2
I20260928 11:03:28.099674 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.099221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47486)
0928 11:03:28.099434 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.099525 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 11:03:28.100750 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.100343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47496)
0928 11:03:28.100524 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.100602 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 11:03:28.100973 27038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:28.125614 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } attempt: 5
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:03:28.139971 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.126588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.3.252.125:39695)
0928 11:03:28.126755 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.126759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.126778 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:03:28.126892 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.126896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.126971 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.127053 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.128032 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.128734 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.130459 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.130939 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.131049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.134798 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.134804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.134807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.134843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.137360 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.138097 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.138101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.138104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.138224 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.138587 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.138590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.138592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.138883 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.139030 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.139422 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.139645 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 11:03:28.139974 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.126351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48326)
0928 11:03:28.126588 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.126612 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:28.126765 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:28.127126 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:28.127132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:28.127633 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:28.127870 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.127883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.128880 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.128886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.130273 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.130291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.131133 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.131140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.131371 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.132012 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:28.132038 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:28.134578 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:28.137514 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.137520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.137524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.137972 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.138335 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.138340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.138344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.138479 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.139002 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:28.139007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:28.139177 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 11:03:28.139293 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 11:03:28.139610 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 11:03:28.141007 26964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:03:28.142086 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.141676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47506)
0928 11:03:28.141862 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.141945 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 11:03:28.142993 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.142686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47522)
0928 11:03:28.142821 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.142900 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 11:03:28.143230 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:28.192200 27038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 3
I20260928 11:03:28.193451 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.192915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47538)
0928 11:03:28.193150 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.193264 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 11:03:28.194433 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 1
I20260928 11:03:28.194626 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.194115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47542)
0928 11:03:28.194351 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.194467 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260928 11:03:28.194870 27038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:28.195369 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.194991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47548)
0928 11:03:28.195146 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.195226 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 11:03:28.196281 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.195995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47554)
0928 11:03:28.196112 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.196179 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 11:03:28.196476 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:28.264695 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 2
I20260928 11:03:28.265803 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.265329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47564)
0928 11:03:28.265546 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.265642 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 11:03:28.266784 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.266465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47568)
0928 11:03:28.266600 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.266682 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 11:03:28.267017 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
W20260928 11:03:28.321333 27045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 11:03:28.321681 27045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 11:03:28.321779 27045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 11:03:28.321916 27045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 11:03:28.329344 27045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 11:03:28.329452 27045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 11:03:28.329509 27045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 11:03:28.329577 27045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 11:03:28.329671 27045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 11:03:28.338104 27038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 4
I20260928 11:03:28.339216 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.338745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47578)
0928 11:03:28.338974 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.339066 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 11:03:28.340263 27045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.252.84:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.252.126:36661,127.3.252.125:39695,127.3.252.124:39997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/krb5kdc/kudu_127.3.252.124.keytab
--principal=kudu/127.3.252.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.252.124:39997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.252.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 11:03:28.340415 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.339954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47594)
0928 11:03:28.340172 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.340289 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 11:03:28.340737 27038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:28.341639 27045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 11:03:28.342950 27045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 11:03:28.354445 27050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:28.354554 27051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 11:03:28.354434 27053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 11:03:28.355162 27045 server_base.cc:1034] running on GCE node
I20260928 11:03:28.355973 27045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 11:03:28.357398 27045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 11:03:28.358609 27045 hybrid_clock.cc:648] HybridClock initialized: now 1790593408358546 us; error 37 us; skew 500 ppm
I20260928 11:03:28.358927 27045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 11:03:28.362075 27045 init.cc:377] Logged in from keytab as kudu/127.3.252.124@KRBTEST.COM (short username kudu)
I20260928 11:03:28.363512 27045 webserver.cc:460] Webserver started at http://127.3.252.124:43263/ using document root <none> and password file <none>
I20260928 11:03:28.364212 27045 fs_manager.cc:359] Metadata directory not provided
I20260928 11:03:28.364313 27045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 11:03:28.364616 27045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 11:03:28.367645 27045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data/instance:
uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5"
format_stamp: "Formatted at 2026-09-28 11:03:28 on dist-test-slave-djjw"
server_key: "cc27e3214cd06fde96a74f897109f03e"
server_key_iv: "d232ba1dd43d67e3e4ae35914d3badf2"
server_key_version: "encryptionkey@0"
I20260928 11:03:28.368438 27045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/wal/instance:
uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5"
format_stamp: "Formatted at 2026-09-28 11:03:28 on dist-test-slave-djjw"
server_key: "cc27e3214cd06fde96a74f897109f03e"
server_key_iv: "d232ba1dd43d67e3e4ae35914d3badf2"
server_key_version: "encryptionkey@0"
I20260928 11:03:28.373229 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 3
I20260928 11:03:28.374006 27045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 11:03:28.374387 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.373865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47596)
0928 11:03:28.374087 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.374192 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 11:03:28.375422 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.375064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47602)
0928 11:03:28.375237 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.375317 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 11:03:28.375638 26964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.252.124:39997: Network error: Client connection negotiation failed: client connection to 127.3.252.124:39997: connect: Connection refused (error 111)
I20260928 11:03:28.377947 27060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 11:03:28.379454 27045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 11:03:28.379614 27045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data,/tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/wal
uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5"
format_stamp: "Formatted at 2026-09-28 11:03:28 on dist-test-slave-djjw"
server_key: "cc27e3214cd06fde96a74f897109f03e"
server_key_iv: "d232ba1dd43d67e3e4ae35914d3badf2"
server_key_version: "encryptionkey@0"
I20260928 11:03:28.379817 27045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 11:03:28.394409 27045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 11:03:28.398566 27045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 11:03:28.398800 27045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 11:03:28.415232 27045 rpc_server.cc:307] RPC server started. Bound to: 127.3.252.124:39997
I20260928 11:03:28.415257 27111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.252.124:39997 every 8 connection(s)
I20260928 11:03:28.416618 27045 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/data/info.pb
I20260928 11:03:28.420394 27112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 11:03:28.421818 4081 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu as pid 27045
I20260928 11:03:28.422014 4081 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktiZmwa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790593056029081-4081-0/minicluster-data/master-2/wal/instance
I20260928 11:03:28.422547 4081 external_mini_cluster.cc:1428] Setting key e60dc90b66fa45f4bc8d65a35b23da14
I20260928 11:03:28.431365 27112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:03:28.448148 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.424664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.3.252.124:39997)
0928 11:03:28.425601 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.425620 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.425672 (+ 52us) server_negotiation.cc:407] Connection header received
0928 11:03:28.428117 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.428158 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.428812 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.429317 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.431438 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.433476 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.435201 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.436016 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.436167 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.440442 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.440451 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.440463 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.440507 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.443734 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.444430 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.444437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.444444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.444631 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.444991 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.444996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.445000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.445730 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.446066 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.446441 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.446897 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":371,"threads_started":1}
I20260928 11:03:28.449568 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.434493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.3.252.126:36661)
0928 11:03:28.434715 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.434721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.435487 (+ 766us) server_negotiation.cc:407] Connection header received
0928 11:03:28.435667 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.435672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.435739 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.435817 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.437302 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.438051 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.439637 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.440070 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.440173 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.443835 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.443842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.443846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.443888 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.446941 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.447619 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.447624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.447627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.447795 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.448160 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.448165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.448168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.448506 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.448632 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.449081 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.449294 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 11:03:28.449693 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.434539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.126:36661 (local address 127.0.0.1:56310)
0928 11:03:28.435350 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.435387 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:28.435545 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:28.435909 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:28.435916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:28.436770 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:28.437106 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.437119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.438217 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.438223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.439494 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.439509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.440318 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.440326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.440491 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.441138 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:28.441196 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:28.443619 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:28.447064 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.447074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.447079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.447504 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.447906 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.447909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.447911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.448056 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.448626 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:28.448632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:28.448820 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 11:03:28.448997 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 11:03:28.449320 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":393,"threads_started":1}
I20260928 11:03:28.451093 27112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.126@KRBTEST.COM
I20260928 11:03:28.464748 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.451811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.125:39695 (local address 127.0.0.1:48330)
0928 11:03:28.452101 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.452117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:28.452218 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:28.452508 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:28.452512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:28.452820 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:28.452958 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.452965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.454100 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.454105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.455157 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.455169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.456026 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.456031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.456265 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.456869 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:28.456885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:28.459060 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:28.462187 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.462192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.462196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.462689 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.463102 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.463107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.463110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.463278 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.463871 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:28.463876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:28.464059 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 11:03:28.464181 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 11:03:28.464447 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260928 11:03:28.464879 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.451813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.3.252.125:39695)
0928 11:03:28.451980 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.451985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.452189 (+ 204us) server_negotiation.cc:407] Connection header received
0928 11:03:28.452314 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.452320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.452398 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.452494 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.453089 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.453965 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.455314 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.455893 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.455987 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.459260 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.459266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.459272 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.459316 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.462054 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.462832 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.462837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.462839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.462986 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.463422 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.463427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.463429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.463732 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.463866 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.464333 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.464512 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 11:03:28.465876 27112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } has no permanent_uuid. Determining permanent_uuid...
I20260928 11:03:28.466346 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.451782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.3.252.126:36661)
0928 11:03:28.451935 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.451940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.451956 (+ 16us) server_negotiation.cc:407] Connection header received
0928 11:03:28.452002 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.452006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.452074 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.452173 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.453472 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.454397 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.455777 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.456292 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.456412 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.460559 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.460564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.460567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.460599 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.463356 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.464032 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.464037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.464039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.464206 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.464590 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.464596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.464608 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.465196 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.465454 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.465800 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.466034 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.124@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.252.125@KRBTEST.COM
I20260928 11:03:28.479888 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.466664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47624)
0928 11:03:28.466820 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.466838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:28.466931 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:28.467817 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:28.467821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:28.468210 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:28.468383 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.468395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.469657 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.469662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.470756 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.470768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.471409 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.471414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.471595 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.472167 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:28.472184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:28.474098 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:28.477480 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.477485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.477490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.477921 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.478237 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.478242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.478246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.478362 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.478920 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:28.478924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:28.479061 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 11:03:28.479166 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 11:03:28.479470 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 11:03:28.479928 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.467321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.3.252.124:39997)
0928 11:03:28.467510 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.467515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.467534 (+ 19us) server_negotiation.cc:407] Connection header received
0928 11:03:28.467605 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.467609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.467690 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.467793 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.468519 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.469501 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.470875 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.471281 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.471385 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.474264 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.474268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.474272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.474299 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.477366 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.478022 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.478027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.478030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.478135 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.478464 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.478467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.478469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.478818 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.478945 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.479272 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.479629 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":71}
I20260928 11:03:28.482640 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.468455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.3.252.125:39695)
0928 11:03:28.468601 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.468605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.468620 (+ 15us) server_negotiation.cc:407] Connection header received
0928 11:03:28.468727 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.468730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.468783 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.468854 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.470412 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.471145 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.472497 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.472907 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.473011 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.477210 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.477215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.477218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.477253 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.479783 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.480635 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.480640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.480642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.480777 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.481204 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.481208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.481211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.481489 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.481754 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.482092 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.482319 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 11:03:28.485340 27112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: Bootstrap starting.
I20260928 11:03:28.487357 4081 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 11:03:28.489192 27112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:28.490474 27112 log.cc:824] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:28.493724 27112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: No bootstrap required, opened a new log
I20260928 11:03:28.490373 26939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56324
I20260928 11:03:28.504360 27112 raft_consensus.cc:348] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.504797 27112 raft_consensus.cc:729] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00654f0a5e145d78c21f4ef6ef0d7f5, State: Initialized, Role: FOLLOWER
I20260928 11:03:28.505282 27112 consensus_queue.cc:260] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.506244 27118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.506587 27118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:28.507463 27112 sys_catalog.cc:564] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 11:03:28.516880 27129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 11:03:28.517010 27129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 11:03:28.514811 27045 master_runner.cc:186] Error getting master registration for 127.3.252.126:36661: OK, instance_id { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" instance_seqno: 1790593407419673 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 11:03:28.525991 27045 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 11:03:28.552830 26964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 4
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.126@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
I20260928 11:03:28.566964 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.553691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.3.252.124:39997)
0928 11:03:28.553960 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.553965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.553988 (+ 23us) server_negotiation.cc:407] Connection header received
0928 11:03:28.554154 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.554158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.554223 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.554300 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.555245 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.555990 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.557739 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.558242 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.558342 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.561682 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.561687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.561690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.561725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.564315 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.564992 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.564996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.564999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.565118 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.565484 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.565488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.565490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.565873 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.566033 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.566398 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.566609 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 11:03:28.566959 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.553627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47626)
0928 11:03:28.553873 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.553896 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:28.554031 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:28.554353 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:28.554359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:28.554876 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:28.555091 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.555104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.556120 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.556126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.557568 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.557586 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.558393 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.558400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.558612 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.559295 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:28.559322 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:28.561496 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:28.564459 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.564465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.564469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.564870 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.565227 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.565232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.565236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.565381 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.565994 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:28.565999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:28.566152 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 11:03:28.566275 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 11:03:28.566579 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 11:03:28.573036 26964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5: Bootstrap starting.
I20260928 11:03:28.577261 26964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:28.578629 26964 log.cc:824] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:28.582192 26964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5: No bootstrap required, opened a new log
I20260928 11:03:28.590149 26964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.590656 26964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ffe625d6ef41dd8d0f5a898a5afca5, State: Initialized, Role: FOLLOWER
I20260928 11:03:28.591252 26964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.592370 27132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.592770 27132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:28.593725 26964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:28.593816 26939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 11:03:28.594043 26939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:28.596486 26939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.597751 26939 leader_election.cc:288] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [CANDIDATE]: Term 1 election: Requested vote from peers 0c41212e1891409e8ed50a85d6221ca2 (127.3.252.125:39695), e00654f0a5e145d78c21f4ef6ef0d7f5 (127.3.252.124:39997)
I20260928 11:03:28.599255 27087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5"
I20260928 11:03:28.599753 27087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:28.603447 27087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ffe625d6ef41dd8d0f5a898a5afca5 in term 1.
I20260928 11:03:28.604223 26915 leader_election.cc:302] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95ffe625d6ef41dd8d0f5a898a5afca5, e00654f0a5e145d78c21f4ef6ef0d7f5; no voters:
I20260928 11:03:28.604773 27132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 11:03:28.611600 27132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [term 1 LEADER]: Becoming Leader. State: Replica: 95ffe625d6ef41dd8d0f5a898a5afca5, State: Running, Role: LEADER
I20260928 11:03:28.612366 27132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.631330 27139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ffe625d6ef41dd8d0f5a898a5afca5. Latest consensus state: current_term: 1 leader_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.631856 27139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:28.633159 27147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 11:03:28.640306 27038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } attempt: 5
W20260928 11:03:28.650882 27066 tablet.cc:2369] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 11:03:28.651687 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.619903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.3.252.124:39997)
0928 11:03:28.620072 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.620078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.634162 (+ 14084us) server_negotiation.cc:407] Connection header received
0928 11:03:28.634559 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.634565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.634648 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.634735 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.636977 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.638145 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.640103 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.640835 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.640949 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.644248 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.644255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.644260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.644303 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.648127 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.649102 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.649108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.649112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.649287 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.649768 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.649774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.649777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.650164 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.650413 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.651034 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.651296 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 11:03:28.659787 26918 tablet.cc:2369] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 11:03:28 dist-test-slave-djjw krb5kdc[26884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.252.125@KRBTEST.COM for kudu/127.3.252.124@KRBTEST.COM
I20260928 11:03:28.670773 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.641019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.252.124:39997 (local address 127.0.0.1:47654)
0928 11:03:28.641402 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 11:03:28.641422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 11:03:28.641527 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 11:03:28.652404 (+ 10877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 11:03:28.652410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 11:03:28.652906 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 11:03:28.653121 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.653134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.654760 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.654766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.656139 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 11:03:28.656158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.657230 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.657237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 11:03:28.657463 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.658280 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 11:03:28.658304 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 11:03:28.661126 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 11:03:28.667809 (+ 6683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.667815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.667820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.668394 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.668864 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 11:03:28.668870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 11:03:28.668874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 11:03:28.669041 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 11:03:28.669694 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 11:03:28.669710 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 11:03:28.669909 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 11:03:28.670055 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 11:03:28.670359 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260928 11:03:28.671937 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.641748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.3.252.124:39997)
0928 11:03:28.652042 (+ 10294us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.652048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.652065 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:03:28.652110 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.652116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.652194 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.653283 (+ 1089us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.653442 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.654553 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.656336 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.657048 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.661299 (+ 4251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.661510 (+ 211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.661516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.661524 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.661557 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.665116 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.668561 (+ 3445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.668566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.668570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.668734 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.669177 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.669181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.669185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.669557 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.671187 (+ 1630us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.671327 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.671586 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10184,"thread_start_us":16753,"threads_started":1}
I20260928 11:03:28.678001 27038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2: Bootstrap starting.
I20260928 11:03:28.686475 27038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2: Neither blocks nor log segments found. Creating new log.
I20260928 11:03:28.687969 27038 log.cc:824] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2: Log is configured to *not* fsync() on all Append() calls
I20260928 11:03:28.695411 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.618931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.3.252.126:36661)
0928 11:03:28.619112 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.619119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.634151 (+ 15032us) server_negotiation.cc:407] Connection header received
0928 11:03:28.634752 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.634758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.634839 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.634911 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.638977 (+ 4066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.640147 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.658755 (+ 18608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.659517 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.659621 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.671868 (+ 12247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.671877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.671882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.671930 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.675974 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.677583 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.677591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.677595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.677846 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.692949 (+ 15103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.692957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.692960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.693543 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.693885 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.694736 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.695012 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 11:03:28.696367 27147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 11:03:28.697291 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:28.619440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.3.252.125:39695)
0928 11:03:28.619619 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.619625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.633367 (+ 13742us) server_negotiation.cc:407] Connection header received
0928 11:03:28.633594 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.633600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.633695 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.633802 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.660233 (+ 26431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.661350 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.666522 (+ 5172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:28.667213 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:28.667313 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:28.678173 (+ 10860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 11:03:28.678182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 11:03:28.678188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 11:03:28.678234 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 11:03:28.682121 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.690037 (+ 7916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.690045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.690049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.690236 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 11:03:28.693336 (+ 3100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 11:03:28.693342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 11:03:28.693345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 11:03:28.693798 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 11:03:28.694122 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:28.695731 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0928 11:03:28.696951 (+ 1220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 11:03:28.709441 27038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2: No bootstrap required, opened a new log
I20260928 11:03:28.721182 27038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.721621 27038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c41212e1891409e8ed50a85d6221ca2, State: Initialized, Role: FOLLOWER
I20260928 11:03:28.722394 27038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } }
I20260928 11:03:28.713938 26897 master_runner.cc:186] Error getting master registration for 127.3.252.125:39695: OK, instance_id { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" instance_seqno: 1790593407916999 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 11:03:28.729888 27158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.730337 27158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:28.732205 27038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 11:03:28.733609 27012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 11:03:28.737865 27087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [term 1 FOLLOWER]: Refusing update from remote peer 95ffe625d6ef41dd8d0f5a898a5afca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:03:28.740283 27013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c41212e1891409e8ed50a85d6221ca2"
I20260928 11:03:28.741775 27012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [term 1 FOLLOWER]: Refusing update from remote peer 95ffe625d6ef41dd8d0f5a898a5afca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 11:03:28.729809 26897 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 11:03:28.739066 27139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260928 11:03:28.749379 27139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 11:03:28.754048 27013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ffe625d6ef41dd8d0f5a898a5afca5 in term 1.
I20260928 11:03:28.783973 27158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ffe625d6ef41dd8d0f5a898a5afca5. Latest consensus state: current_term: 1 leader_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.784333 27158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:28.816180 27118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ffe625d6ef41dd8d0f5a898a5afca5. Latest consensus state: current_term: 1 leader_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.816514 27118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:28.821547 27161 mvcc.cc:205] Tried to move back new op lower bound from 7334270602123161600 to 7334270601682821120. Current Snapshot: MvccSnapshot[applied={T|T < 7334270602123161600}]
I20260928 11:03:28.823249 27139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.823555 27139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:28.825958 27139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.826210 27139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5 [sys.catalog]: This master's current role is: LEADER
I20260928 11:03:28.829380 27147 catalog_manager.cc:1347] Generated new cluster ID: e6d5f0f3b00440ef90fbcd4d5517fc6d
I20260928 11:03:28.829483 27147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 11:03:28.839057 27118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.839367 27118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:28.842626 27158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ffe625d6ef41dd8d0f5a898a5afca5" member_type: VOTER last_known_addr { host: "127.3.252.126" port: 36661 } } peers { permanent_uuid: "0c41212e1891409e8ed50a85d6221ca2" member_type: VOTER last_known_addr { host: "127.3.252.125" port: 39695 } } peers { permanent_uuid: "e00654f0a5e145d78c21f4ef6ef0d7f5" member_type: VOTER last_known_addr { host: "127.3.252.124" port: 39997 } } }
I20260928 11:03:28.842926 27158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 11:03:28.874184 27147 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 11:03:28.876194 27147 catalog_manager.cc:1504] Loading token signing keys...
I20260928 11:03:28.900727 27176 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 11:03:28.909121 27147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95ffe625d6ef41dd8d0f5a898a5afca5: Generated new TSK 0
I20260928 11:03:28.910418 27147 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 11:03:28.985818 26966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:28.976238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.3.252.126:36661)
0928 11:03:28.976704 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.976711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.983904 (+ 7193us) server_negotiation.cc:407] Connection header received
0928 11:03:28.984086 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.984092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.984203 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.984339 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.985514 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56336: BlockingRecv error: recv got EOF from 127.0.0.1:56336 (error 108)
Metrics: {"server-negotiator.queue_time_us":304}
W20260928 11:03:28.986353 27115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:28.980653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.3.252.124:39997)
0928 11:03:28.980851 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.980859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.984480 (+ 3621us) server_negotiation.cc:407] Connection header received
0928 11:03:28.984790 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.984796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.984878 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.984955 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.986021 (+ 1066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47664: BlockingRecv error: recv got EOF from 127.0.0.1:47664 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 11:03:28.988539 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:28.976015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.3.252.125:39695)
0928 11:03:28.976508 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.976516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.983550 (+ 7034us) server_negotiation.cc:407] Connection header received
0928 11:03:28.987270 (+ 3720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.987274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.987357 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.987441 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.988332 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48364: BlockingRecv error: recv got EOF from 127.0.0.1:48364 (error 108)
Metrics: {"server-negotiator.queue_time_us":319}
W20260928 11:03:28.994869 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:28.992710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.3.252.125:39695)
0928 11:03:28.992927 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.992934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:28.993441 (+ 507us) server_negotiation.cc:407] Connection header received
0928 11:03:28.993672 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:28.993676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:28.993783 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:28.993905 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:28.994609 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48376: BlockingRecv error: recv got EOF from 127.0.0.1:48376 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 11:03:29.003650 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:28.999148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.3.252.125:39695)
0928 11:03:28.999335 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:28.999341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.002342 (+ 3001us) server_negotiation.cc:407] Connection header received
0928 11:03:29.002544 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.002548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.002620 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.002702 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.003427 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48384: BlockingRecv error: recv got EOF from 127.0.0.1:48384 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 11:03:29.026871 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.021752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.3.252.125:39695)
0928 11:03:29.021995 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.022002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.025465 (+ 3463us) server_negotiation.cc:407] Connection header received
0928 11:03:29.025664 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.025669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.025759 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.025884 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.026606 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48398: BlockingRecv error: recv got EOF from 127.0.0.1:48398 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260928 11:03:29.035691 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.033614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.3.252.125:39695)
0928 11:03:29.033889 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.033895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.034395 (+ 500us) server_negotiation.cc:407] Connection header received
0928 11:03:29.034591 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.034596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.034667 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.034748 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.035449 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48412: BlockingRecv error: recv got EOF from 127.0.0.1:48412 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260928 11:03:29.049294 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.047104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.3.252.125:39695)
0928 11:03:29.047383 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.047389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.047890 (+ 501us) server_negotiation.cc:407] Connection header received
0928 11:03:29.048109 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.048115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.048216 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.048309 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.049067 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48420: BlockingRecv error: recv got EOF from 127.0.0.1:48420 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 11:03:29.071143 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.069089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.3.252.125:39695)
0928 11:03:29.069313 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.069319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.069853 (+ 534us) server_negotiation.cc:407] Connection header received
0928 11:03:29.070038 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.070042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.070118 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.070203 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.070912 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48422: BlockingRecv error: recv got EOF from 127.0.0.1:48422 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 11:03:29.111243 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.109361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.3.252.125:39695)
0928 11:03:29.109567 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.109574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.109809 (+ 235us) server_negotiation.cc:407] Connection header received
0928 11:03:29.110049 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.110054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.110135 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.110217 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.110966 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48432: BlockingRecv error: recv got EOF from 127.0.0.1:48432 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 11:03:29.180521 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.178745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.3.252.125:39695)
0928 11:03:29.178957 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.178963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.179195 (+ 232us) server_negotiation.cc:407] Connection header received
0928 11:03:29.179371 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.179377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.179481 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.179614 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.180293 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48448: BlockingRecv error: recv got EOF from 127.0.0.1:48448 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 11:03:29.313648 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.311838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.3.252.125:39695)
0928 11:03:29.312004 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.312008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.312315 (+ 307us) server_negotiation.cc:407] Connection header received
0928 11:03:29.312498 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.312502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.312579 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.312655 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.313385 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48454: BlockingRecv error: recv got EOF from 127.0.0.1:48454 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 11:03:29.519080 27129 catalog_manager.cc:1259] Loaded cluster ID: e6d5f0f3b00440ef90fbcd4d5517fc6d
I20260928 11:03:29.519196 27129 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: loading cluster ID for follower catalog manager: success
I20260928 11:03:29.521708 27129 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: acquiring CA information for follower catalog manager: success
I20260928 11:03:29.523599 27129 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e00654f0a5e145d78c21f4ef6ef0d7f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 11:03:29.575105 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 11:03:29.573171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.3.252.125:39695)
0928 11:03:29.573359 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:29.573363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:29.573571 (+ 208us) server_negotiation.cc:407] Connection header received
0928 11:03:29.573862 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:29.573869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:29.573963 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:29.574062 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 11:03:29.574859 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48456: BlockingRecv error: recv got EOF from 127.0.0.1:48456 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 11:03:29.905488 27176 catalog_manager.cc:1259] Loaded cluster ID: e6d5f0f3b00440ef90fbcd4d5517fc6d
I20260928 11:03:29.905629 27176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2: loading cluster ID for follower catalog manager: success
I20260928 11:03:29.908567 27176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2: acquiring CA information for follower catalog manager: success
I20260928 11:03:29.910921 27176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c41212e1891409e8ed50a85d6221ca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 11:03:30.097975 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:30.090858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.3.252.125:39695)
0928 11:03:30.091637 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:30.091641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:30.091659 (+ 18us) server_negotiation.cc:407] Connection header received
0928 11:03:30.091723 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:30.091727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:30.091886 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:30.091987 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:30.092548 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.093465 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.095476 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.095949 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.096052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:30.096707 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:30.097240 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:30.097352 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:30.097500 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 11:03:30.097581 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":513,"threads_started":1}
I20260928 11:03:30.111120 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:30.104719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.3.252.124:39997)
0928 11:03:30.105321 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:30.105325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:30.105342 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:03:30.105438 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:30.105442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:30.105595 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:30.105741 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:30.106207 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.106912 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.108827 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.109285 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.109378 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:30.109944 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:30.110515 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:30.110635 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:30.110763 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 11:03:30.110834 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":387,"threads_started":1}
I20260928 11:03:30.140154 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:30.118293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.3.252.124:39997)
0928 11:03:30.133740 (+ 15447us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:30.133748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:30.133776 (+ 28us) server_negotiation.cc:407] Connection header received
0928 11:03:30.133964 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:30.133970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:30.134062 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:30.134123 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:30.134861 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.135917 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.137963 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.138566 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.138663 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:30.139330 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:30.139472 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:30.139536 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:30.139778 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 11:03:30.139870 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15276}
I20260928 11:03:30.141304 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:30.118008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.3.252.125:39695)
0928 11:03:30.118364 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:30.118372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:30.134591 (+ 16219us) server_negotiation.cc:407] Connection header received
0928 11:03:30.134794 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:30.134800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:30.134902 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:30.134984 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:30.135747 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.136864 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.139100 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.139737 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.139824 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:30.140446 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:30.140583 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:30.140649 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:30.140877 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 11:03:30.140971 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260928 11:03:30.142298 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 11:03:30.118523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.3.252.126:36661)
0928 11:03:30.135108 (+ 16585us) server_negotiation.cc:206] Beginning negotiation
0928 11:03:30.135114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 11:03:30.135131 (+ 17us) server_negotiation.cc:407] Connection header received
0928 11:03:30.135247 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 11:03:30.135252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 11:03:30.135476 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 11:03:30.136228 (+ 752us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 11:03:30.136394 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.137406 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.139498 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 11:03:30.140099 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 11:03:30.140647 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 11:03:30.140835 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 11:03:30.141269 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 11:03:30.141898 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0928 11:03:30.141948 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 11:03:30.142029 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16453,"thread_start_us":20391,"threads_started":1}
I20260928 11:03:30.147027 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26897
I20260928 11:03:30.148263 26959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:30.257656 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26897
W20260928 11:03:30.275240 26990 connection.cc:535] server connection from 127.0.0.1:48326 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 11:03:30.275731 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 26971
I20260928 11:03:30.276796 27033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:30.383404 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 26971
I20260928 11:03:30.401023 4081 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu pid 27045
I20260928 11:03:30.402237 27107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 11:03:30.507089 4081 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktiZmwa/build/asan/bin/kudu with pid 27045
2026-09-28T11:03:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4818 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9115 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (354442 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST